Customized Presentation of Items on Electronic Visual Displays in Retail Stores Based on Condition of Products

ABSTRACT

Methods, systems, and computer-readable media are provided for customized presentation of items on electronic visual displays in retail stores. In one implementation, the method may comprise: obtaining an image of products in a retail store captured using at least one image sensor; analyzing the image to determine a condition of products of a particular product type; based on the determined condition of the products of the particular product type, selecting at least one display parameter for a particular item; and using the selected at least one display parameter to display the particular item on an electronic visual display in the retail store.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority of U.S. ProvisionalApplication No. 62/876,685, filed Jul. 21, 2019. The foregoingapplication is incorporated herein by reference in its entirety.

BACKGROUND I. Technical Field

The present disclosure relates generally to systems, methods, anddevices for providing information in retail stores, and morespecifically to systems, methods, and devices for capturing, providinginformation on electronic visual displays in retail stores.

II. Background Information

Shopping in stores is a prevalent part of modern daily life. Storeowners (also known as “retailers”) stock a wide variety of products onstore shelves and add associated labels and promotions to the storeshelves. Typically, retailers have a set of processes and instructionsfor providing information in retail stores. The source of some of theseinstructions may include contractual obligations and other preferencesrelated to the retailer methodology for providing information. Moreover,providing selected information may drive higher sales, improvecustomers' experience, and enhance in-store execution. Nowadays, manyretailers and suppliers send people to stores to personally monitor andcontrol the provided information. Such a monitoring technique, however,may be inefficient and may result in nonuniform compliance amongretailers relative to various product-related guidelines. This techniquemay also result in significant gaps in compliance, as it does not allowfor continuous monitoring of dynamically changing product displays. Toincrease productivity, among other potential benefits, there is atechnological need to provide a dynamic solution that will automaticallyprovide selected information.

The disclosed devices and methods are directed to providing new ways forproviding information in retail stores.

SUMMARY

Embodiments consistent with the present disclosure provide methods,systems, and computer-readable media are provided for providinginformation on electronic visual displays in retail stores. In oneimplementation, a door for a retail storage container may include one ormore electronic visual displays. In one implementation, the electronicvisual display may be connected to a shelf in the retail store.

In some embodiments, methods, systems, and computer-readable media areprovided for controlling information displayed on an electronic visualdisplay that is part of a door for a retail storage container. In someexamples, a door for a retail storage container is provided.

In some embodiments, a door for a retail storage container may compriseat least a first part that may be configured to face customers when thedoor is closed and a second part that may be configured to face theinternal side of the retail storage container when the door is closed.The second part may comprise at least an electronic visual displayconfigured to display information, and at least part of the electronicvisual display may be configured to be visible to the customers at leastwhen the door is open at a selected angle. In one example, the at leastpart of the electronic visual display may be configured to be hiddenfrom the customers when the door is closed. In one example, the retailstorage container may be a refrigerator unit. In one example, thedisplayed information may be based on a person facing the retail storagecontainer. In one example, the displayed information may be based ondata related to products stored in the retail storage container. In oneexample, the displayed information may be based on a label positioned inthe retail storage container. In one example, the retail storagecontainer may comprise a shelf, a plurality of sensors may be positionedon the shelf and may be configured to be positioned between the shelfand products positioned on the shelf, and the displayed information maybe based on an analysis of data captured using the plurality of sensors.In one example, the retail storage container may comprise a shelf, andthe displayed information may be based on an analysis of weight datacaptured using a weight sensor, the weight sensor may be configured tomeasure a weight of at least one product placed on the shelf. In oneexample, an indication of a state of the door may be received, inresponse to a first state of the door, the electronic visual display maybe caused to display the information, and in response to a second stateof the door, causing the electronic visual display to display theinformation may be forgone. In one example, an indication of whether thedoor is open may be received, and an adjustment to a power scheme of theelectronic visual display may be caused based on the receivedindication.

In some examples, the retail storage container may comprise an imagesensor, and the second part may further comprise a mirror that may beconfigured to reflect towards the image sensor an image of at least aportion of an internal part of the retail storage container. Forexample, the displayed information may be based on an analysis of theimage reflected by the mirror and digitally captured using the imagesensor. In another example, the image sensor may be configured tocapture an image of a person facing the retail storage container whenthe door is open. In yet another example, the retail storage containermay comprise a shelf, and the mirror may be configured to reflecttowards the image sensor an image of at least part of the shelf and ofan area above the shelf In an additional example, an indication that thedoor is closed may be received, and in response to the receivedindication, the image sensor may be caused to capture at least oneimage.

In some examples, the second part may further comprise an image sensorthat may be configured to capture at least one image of at least aportion of an internal part of the retail storage container. Forexample, the displayed information may be based on an analysis of the atleast one image. In another example, the image sensor may be configuredto capture an image of a person facing the retail storage container whenthe door is open. In yet another example, the retail storage containermay comprise a shelf, and the image sensor may be configured to capturean image of at least part of the shelf and of an area above the shelf Inan additional example, an indication that the door is closed may bereceived, and in response to the received indication, the image sensormay be caused to capture the at least one image.

In some embodiments, methods, systems, and computer-readable media areprovided for controlling information displayed on a transparentelectronic display that is part of a door for a retail storagecontainer.

In some embodiments, an indication of at least one position associatedwith a first product type in the retail storage container may bereceived, an indication of at least one position associated with asecond product type in the retail storage container, the second producttype differs from the first product type may be received, the indicationof the at least one position associated with the first product type maybe used to select a first region of the transparent electronic display,the indication of the at least one position associated with the secondproduct type may be used to select a second region of the transparentelectronic display, the second region differs from the first region,visual information related to the first product type may be displayed onthe first region of the transparent electronic display, and visualinformation related to the second product type may be displayed on thesecond region of the transparent electronic display. In one example, theselection of the first region of the transparent electronic display maybe configured to cause at least part of the displayed visual informationrelated to the first product type to appear over at least part of the atleast one position associated with the first product type when viewedfrom a particular viewing point, and the selection of the second regionof the transparent electronic display may be configured to cause atleast part of the displayed visual information related to the secondproduct type to appear over at least part of the at least one positionassociated with the second product type when viewed from the particularviewing point. In one example, the selection of the first region of thetransparent electronic display and the selection of the second region ofthe transparent electronic display may be based on a person facing theretail storage container. In one example, the at least one positionassociated with the first product type may include a position of thefirst product type in a planogram, and the at least one positionassociated with the second product type may include a position of thesecond product type in the planogram. In one example, the indication ofthe at least one position associated with the first product type may bebased on an analysis of at least one image of products placed in theretail storage container. In one example, the retail storage containermay comprise a shelf, a plurality of sensors may be positioned on theshelf and may be configured to be positioned between the shelf andproducts positioned on the shelf, and the indication of the at least oneposition associated with the first product type may be based on ananalysis of data captured using the plurality of sensors. In oneexample, the retail storage container may comprise a shelf, and theindication of the at least one position associated with the firstproduct type may be based on an analysis of weight data captured usingthe weight sensor, the weight sensor may be configured to measure aweight of at least one product placed on the shelf. In one example, theat least one position associated with the first product type may includea position of products of the first product type in the retail storagecontainer. In one example, the at least one position associated with thefirst product type may include a position of a label corresponding tothe first product type in the retail storage container. In one example,the at least one position associated with the first product type mayinclude a position of an empty space dedicated to the first product typein the retail storage container. In one example, the at least oneposition associated with the first product type may include a positionat which products of the first product type were previously placed inthe retail storage container and at which products of the first producttype are not currently placed. In one example, the displayed visualinformation related to the first product type may be based on ananalysis of at least one image of products placed in the retail storagecontainer. In one example, the retail storage container may comprise ashelf, a plurality of sensors may be positioned on the shelf and may beconfigured to be positioned between the shelf and products positioned onthe shelf, and the displayed visual information related to the firstproduct type may be based on an analysis of data captured using theplurality of sensors. In one example, the retail storage container maycomprise a shelf, and the displayed visual information related to thefirst product type may be based on an analysis of weight data capturedusing the weight sensor, the weight sensor may be configured to measurea weight of at least one product placed on the shelf. In one example,the displayed visual information related to the first product type maybe based on a state of the door. In one example, the displayed visualinformation related to the first product type may be based on an amountof products of the first product type placed in the retail storagecontainer. In one example, an amount of products of the first producttype in the retail storage container may be obtained, the amount ofproducts of the first product type in the retail storage container maybe compared with a selected threshold, in response to a first result ofthe comparison, first visual information related to the first producttype may be displayed, and in response to a second result of thecomparison, second visual information related to the first product typemay be displayed, the second visual information may differ from thefirst visual information. In one example, the displayed visualinformation related to the first product type may be based on facings ofthe first product type in the retail storage container. In one example,the displayed visual information related to the first product type maybe based on information presented on a label corresponding to the firstproduct type. In one example, the displayed visual information relatedto the first product type may be based on a price corresponding to thefirst product type. In one example, the displayed visual informationrelated to the first product type may be based on the selected firstregion of the transparent electronic display. In one example, thedisplayed visual information related to the first product type may bebased on the at least one position associated with the first producttype in the retail storage container. In one example, the displayedvisual information related to the first product type may be based on aperson facing the retail storage container. In one example, thedisplayed visual information related to the first product type mayinclude an indication of a need to restock the first product type in theretail storage container.

In some embodiments, methods, systems, and computer-readable media areprovided for selecting items for presentation on electronic visualdisplays in retail stores. In some embodiments, methods, systems, andcomputer-readable media are provided for customized presentation ofitems on electronic visual displays in retail stores.

In some embodiments, a plurality of images of products in a retail storecaptured using at least one image sensor may be obtained. The pluralityof images may comprise at least a first image corresponding to a firstpoint in time and a second image corresponding to a second point intime, the first point in time is earlier than the second point in time.Further, in some examples, the first image may be analyzed to determinewhether products of a particular product type are available at the firstpoint in time, and the second image may be analyzed to determine whetherproducts of the particular product type are available at the secondpoint in time. Further, in some examples, based on the determination ofwhether products of the particular product type are available at thefirst point in time and the determination of whether products of theparticular product type are available at the second point in time, itmay be selected whether to display a particular item on an electronicvisual display in the retail store. Further, in some examples, inresponse to a selection to display the particular item, causing theelectronic visual display to display the particular item, and inresponse to a selection not to display the particular item, forgoingcausing the electronic visual display to display the particular item.

In one example, in response to a determination that products of theparticular product type are missing at the first point in time and adetermination that products of the particular product type are missingat the second point in time, it may be selected not to display theparticular item on the electronic visual display in the retail store,and in response to at least one of a determination that products of theparticular product type are available at the first point in time and adetermination that products of the particular product type are availableat the second point in time, it may be selected to display theparticular item on the electronic visual display in the retail store.

In one example, in response to a determination that products of theparticular product type are missing at the first point in time and adetermination that products of the particular product type are missingat the second point in time, it may be selected to display theparticular item on the electronic visual display in the retail store,and in response to at least one of a determination that products of theparticular product type are available at the first point in time and adetermination that products of the particular product type are availableat the second point in time, it may be selected not to display theparticular item on the electronic visual display in the retail store.

In some examples, the plurality of images may comprise a preceding imagecorresponding to a preceding point in time, the preceding image may beanalyzed to determine whether products of the particular product typeare available at the preceding point in time, and the selection ofwhether to display the particular item on the electronic visual displayin the retail store may be further based on the determination of whetherproducts of the particular product type are available at the precedingpoint in time. In one example, in response to a determination thatproducts of the particular product type are missing at the precedingpoint in time, a determination that products of the particular producttype are available at the first point in time and a determination thatproducts of the particular product type are missing at the second pointin time, it may be selected not to display the particular item on theelectronic visual display in the retail store, and in response to adetermination that products of the particular product type are availableat the preceding point in time, the determination that products of theparticular product type are available at the first point in time and thedetermination that products of the particular product type are missingat the second point in time, it may be selected to display theparticular item on the electronic visual display in the retail store. Inone example, in response to a determination that products of theparticular product type are missing at the preceding point in time, adetermination that products of the particular product type are missingat the first point in time and a determination that products of theparticular product type are missing at the second point in time, it maybe selected not to display the particular item on the electronic visualdisplay in the retail store, and in response to at least one of adetermination that products of the particular product type are availableat the preceding point in time, a determination that products of theparticular product type are available at the first point in time and thedetermination that products of the particular product type are availableat the second point in time, it may be selected to display theparticular item on the electronic visual display in the retail store. Inone example, in response to a determination that products of theparticular product type are missing at the preceding point in time, adetermination that products of the particular product type are missingat the first point in time and a determination that products of theparticular product type are missing at the second point in time, it maybe selected to display the particular item on the electronic visualdisplay in the retail store, and in response to at least one of adetermination that products of the particular product type are availableat the preceding point in time, a determination that products of theparticular product type are available at the first point in time and thedetermination that products of the particular product type are availableat the second point in time, it may be selected not to display theparticular item on the electronic visual display in the retail store. Inone example, in response to a determination that products of theparticular product type are missing at the preceding point in time, adetermination that products of the particular product type are availableat the first point in time and a determination that products of theparticular product type are missing at the second point in time, it maybe selected to display the particular item on the electronic visualdisplay in the retail store, and in response to at least one of adetermination that products of the particular product type are availableat the preceding point in time and a determination that products of theparticular product type are available at the second point in time, itmay be selected not to display the particular item on the electronicvisual display in the retail store.

In one example, the selection of whether to display the particular itemon the electronic visual display in the retail store may be furtherbased on an elapsed time between the first point in time and the secondpoint in time. In one example, the selection of whether to display theparticular item on the electronic visual display in the retail store maybe further based on an elapsed time since the second point in time. Inone example, the selection of whether to display the particular item onthe electronic visual display in the retail store may be further basedon information related to a person in a vicinity of the electronicvisual display. In one example, the selection of whether to display theparticular item on the electronic visual display in the retail store maybe further based on a time of day.

In one example, the electronic visual display may be connected to ashelf in the retail store. In one example, the electronic visual displaymay be connected to a door of a retail storage container in the retailstore. In one example, the electronic visual display may be part of apersonal device of a store associate. In one example, the electronicvisual display may be part of a personal device of a customer.

In one example, data captured at the first point in time using aplurality of sensors positioned on a shelf in the retail store that maybe configured to be positioned between the shelf and products positionedon the shelf may be obtained, data captured at the second point in timeusing the plurality of sensors may be obtained, the determination ofwhether products of the particular product type are available at thefirst point in time may be based on an analysis of the data captured atthe first point in time using the plurality of sensors, and thedetermination of whether products of the particular product type areavailable at the second point in time may be based on an analysis of thedata captured at the second point in time using the plurality ofsensors.

In one example, weight data captured at the first point in time using aweight sensor corresponding to at least part of a shelf in the retailstore may be obtained, weight data captured at the second point in timeusing the weight sensor may be obtained, the determination of whetherproducts of the particular product type are available at the first pointin time may be based on an analysis of the weight data captured at thefirst point in time using the weight sensor, and the determination ofwhether products of the particular product type are available at thesecond point in time may be based on an analysis of the weight datacaptured at the second point in time using the weight sensor.

In some embodiments, a plurality of images of products in a retail storecaptured using at least one image sensor may be obtained. The pluralityof images may comprise at least a first image corresponding to a firstpoint in time and a second image corresponding to a second point intime, the first point in time is earlier than the second point in time.Further, in some examples, the first image may be analyzed to determinewhether products of a particular product type are available at the firstpoint in time, and the second image may be analyzed to determine whetherproducts of the particular product type are available at the secondpoint in time. Further, in some examples, based on the determination ofwhether products of the particular product type are available at thefirst point in time and the determination of whether products of theparticular product type are available at the second point in time, atleast one display parameter for a particular item may be selected.Further, in some examples, the selected at least one display parametermay be used to display the particular item on an electronic visualdisplay in the retail store.

In one example, the at least one display parameter may include a displaysize for the particular item. In one example, the at least one displayparameter may include a motion pattern for the particular item. In oneexample, the at least one display parameter may include a displayposition on the electronic visual display for the particular item. Inone example, the at least one display parameter may include a colorscheme for the particular item.

In one example, the plurality of images may comprise a preceding imagecorresponding to a preceding point in time, the preceding image may beanalyzed to determine whether products of the particular product typeare available at the preceding point in time, and the selection of theat least one display parameter for the particular item may be furtherbased on the determination of whether products of the particular producttype are available at the preceding point in time.

In one example, the selection of the at least one display parameter forthe particular item may be further based on an elapsed time between thefirst point in time and the second point in time. In one example, theselection of the at least one display parameter for the particular itemmay be further based on an elapsed time since the second point in time.In one example, the selection of the at least one display parameter forthe particular item may be further based on information related to aperson in a vicinity of the electronic visual display. In one example,the selection of the at least one display parameter for the particularitem may be further based on a time of day.

In one example, the electronic visual display may be connected to ashelf in the retail store. In one example, the electronic visual displaymay be connected to a door of a retail storage container in the retailstore. In one example, the electronic visual display may be part of apersonal device of a store associate. In one example, the electronicvisual display may be part of a personal device of a customer.

In one example, data captured at the first point in time using aplurality of sensors positioned on a shelf in the retail store that maybe configured to be positioned between the shelf and products positionedon the shelf may be obtained, data captured at the second point in timeusing the plurality of sensors may be obtained, the determination ofwhether products of the particular product type are available at thefirst point in time may be based on an analysis of the data captured atthe first point in time using the plurality of sensors, and thedetermination of whether products of the particular product type areavailable at the second point in time may be based on an analysis of thedata captured at the second point in time using the plurality ofsensors.

In one example, weight data captured at the first point in time using aweight sensor corresponding to at least part of a shelf in the retailstore may be obtained, weight data captured at the second point in timeusing the weight sensor may be obtained, the determination of whetherproducts of the particular product type are available at the first pointin time may be based on an analysis of the weight data captured at thefirst point in time using the weight sensor, and the determination ofwhether products of the particular product type are available at thesecond point in time may be based on an analysis of the weight datacaptured at the second point in time using the weight sensor.

In some embodiments, an image of products in a retail store capturedusing at least one image sensor may be obtained, and the image may beanalyzed to determine a condition of products of a particular producttype. Further, in some examples, based on the determined condition ofthe products of the particular product type, selecting whether todisplay a particular item on an electronic visual display in the retailstore. Further, in some examples, in response to a selection to displaythe particular item, the electronic visual display may be caused todisplay the particular item, and in response to a selection not todisplay the particular item, causing the electronic visual display todisplay the particular item may be forgone.

In one example, the particular item may include an indication of theparticular product type. In one example, the particular item may includean indication of the determined condition of the products of theparticular product type. In one example, the selection of whether todisplay the particular item on the electronic visual display in theretail store may be further based on an elapsed time since the capturingof the image. In one example, the selection of whether to display theparticular item on the electronic visual display in the retail store maybe further based on a time of day. In one example, the selection ofwhether to display the particular item on the electronic visual displayin the retail store may be further based on information related to aperson in a vicinity of the electronic visual display. In one example,the electronic visual display may be connected to a shelf in the retailstore. In one example, the electronic visual display may be connected toa door of a retail storage container in the retail store. In oneexample, the electronic visual display may be part of a personal deviceof a store associate. In one example, the electronic visual display maybe part of a personal device of a customer. In one example, datacaptured using a plurality of sensors positioned on a shelf in theretail store that may be configured to be positioned between the shelfand products positioned on the shelf may be obtained, and thedetermination of the condition of the products of the particular producttype may be further based on an analysis of the data captured using theplurality of sensors.

In some examples, a preceding image of products in a retail storecaptured using the at least one image sensor at a preceding point intime before the capturing time of the image may be obtained, thepreceding image may be analyzed to determine a preceding condition ofthe products of the particular product type at the preceding point intime, and the selection of whether to display the particular item on theelectronic visual display in the retail store may be further based onthe determined preceding condition. For example, the determinedpreceding condition may be compared with the determined condition, andthe selection of whether to display the particular item on theelectronic visual display in the retail store may be based on a resultof the comparison. In another example, the determined precedingcondition and the determined condition may be used to predict a futurecondition of products of the particular product type at a later point intime after the capturing time of the image, and the selection of whetherto display the particular item on the electronic visual display in theretail store may be based on the predicted future condition.

In one example, in response to a determination that the condition of theproducts of the particular product type is a good condition, it may beselected to display the particular item on the electronic visual displayin the retail store, and in response to a determination that thecondition of the products of the particular product type is a badcondition, it may be selected not to display the particular item on theelectronic visual display in the retail store. In one example, inresponse to a determination that the condition of the products of theparticular product type is a bad condition, it may be selected todisplay the particular item on the electronic visual display in theretail store, and in response to a determination that the condition ofthe products of the particular product type is a good condition, it maybe selected not to display the particular item on the electronic visualdisplay in the retail store. In one example, in response to adetermination that the condition of the products of the particularproduct type is a condition that requires maintenance, it may beselected to display the particular item on the electronic visual displayin the retail store, and in response to a determination that thecondition of the products of the particular product type is a conditionthat do not require maintenance, it may be selected not to display theparticular item on the electronic visual display in the retail store. Inone example, the image may be analyzed to determine a condition of theproducts of a second product type, the second product type differs fromthe particular product type, and the selection of whether to display theparticular item on the electronic visual display in the retail store maybe further based on the determined condition of the products of thesecond product type.

In some embodiments, an image of products in a retail store capturedusing at least one image sensor may be obtained, and the image may beanalyzed to determine a condition of products of a particular producttype. Further, in some examples, based on the determined condition ofthe products of the particular product type, at least one displayparameter for a particular item may be selected, and the selected atleast one display parameter may be used to display the particular itemon an electronic visual display in the retail store.

In one example, the at least one display parameter may include a displaysize for the particular item. In one example, the at least one displayparameter may include a motion pattern for the particular item. In oneexample, the at least one display parameter may include a displayposition on the electronic visual display for the particular item. Inone example, the at least one display parameter may include a colorscheme for the particular item. In one example, the selection of the atleast one display parameter for the particular item may be further basedon an elapsed time since the capturing of the image. In one example, theselection of the at least one display parameter for the particular itemmay be further based on a time of day. In one example, the selection ofthe at least one display parameter for the particular item may befurther based on information related to a person in a vicinity of theelectronic visual display.

In some examples, a preceding image of products in a retail storecaptured using the at least one image sensor at a preceding point intime before the capturing time of the image may be obtained, thepreceding image may be analyzed to determine a preceding condition ofthe products of the particular product type at the preceding point intime, and the selection of the at least one display parameter for theparticular item may be further based on the determined precedingcondition. For example, the determined preceding condition may becompared with the determined condition, and the selection of the atleast one display parameter for the particular item may be based on aresult of the comparison. In another example, the determined precedingcondition and the determined condition may be used to predict a futurecondition of products of the particular product type at a later point intime after the capturing time of the image, and the selection of the atleast one display parameter for the particular item may be based on thepredicted future condition.

In one example, the electronic visual display may be connected to ashelf in the retail store. In one example, the electronic visual displaymay be connected to a door of a retail storage container in the retailstore. In one example, the electronic visual display may be part of apersonal device of a store associate. In one example, the electronicvisual display may be part of a personal device of a customer. In oneexample, data captured using a plurality of sensors positioned on ashelf in the retail store that may be configured to be positionedbetween the shelf and products positioned on the shelf may be obtained,and the determination of the condition of the products of the particularproduct type may be based on an analysis of the data captured using theplurality of sensors. In one example, the image may be analyzed todetermine an indicator of urgency of the required maintenance, and theselection of the at least one display parameter for the particular itemmay be based on the determined indicator of urgency. In one example, theimage may be analyzed to determine a condition of the products of asecond product type, the second product type differs from the particularproduct type, and the selection of the at least one display parameterfor the particular item may be based on the determined condition of theproducts of the second product type.

Consistent with other disclosed embodiments, non-transitorycomputer-readable medium including instructions that when executed by aprocessor may cause the processor to perform any of the methodsdescribed herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is an illustration of an exemplary system for analyzinginformation collected from a retail store.

FIG. 2 is a block diagram that illustrates some of the components of animage processing system, consistent with the present disclosure.

FIG. 3 is a block diagram that illustrates an exemplary embodiment of acapturing device, consistent with the present disclosure.

FIG. 4A is a schematic illustration of an example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 4B is a schematic illustration of another example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 4C is a schematic illustration of another example configuration forcapturing image data in a retail store, consistent with the presentdisclosure.

FIG. 5A is an illustration of an example system for acquiring images ofproducts in a retail store, consistent with the present disclosure.

FIG. 5B is an illustration of a shelf-mounted camera unit included in afirst housing of the example system of FIG. 5A, consistent with thepresent disclosure.

FIG. 5C is an exploded view illustration of a processing unit includedin a second housing of the example system of FIG. 5A, consistent withthe present disclosure.

FIG. 6A is a top view representation of an aisle in a retail store withmultiple image acquisition systems deployed thereon for acquiring imagesof products, consistent with the present disclosure.

FIG. 6B is a perspective view representation of part of a retailshelving unit with multiple image acquisition systems deployed thereonfor acquiring images of products, consistent with the presentdisclosure.

FIG. 6C provides a diagrammatic representation of how the exemplarydisclosed image acquisition systems may be positioned relative to retailshelving to acquire product images, consistent with the presentdisclosure.

FIG. 7A provides a flowchart of an exemplary method for acquiring imagesof products in retail store, consistent with the present disclosure.

FIG. 7B provides a flowchart of a method for acquiring images ofproducts in retail store, consistent with the present disclosure.

FIG. 8A is a schematic illustration of an example configuration fordetecting products and empty spaces on a store shelf, consistent withthe present disclosure.

FIG. 8B is a schematic illustration of another example configuration fordetecting products and empty spaces on a store shelf, consistent withthe present disclosure.

FIG. 9 is a schematic illustration of example configurations fordetection elements on store shelves, consistent with the presentdisclosure.

FIG. 10A illustrates an exemplary method for monitoring planogramcompliance on a store shelf, consistent with the present disclosure.

FIG. 10B is illustrates an exemplary method for triggering imageacquisition based on product events on a store shelf, consistent withthe present disclosure.

FIG. 11A is a schematic illustration of an example output for a marketresearch entity associated with the retail store, consistent with thepresent disclosure.

FIG. 11B is a schematic illustration of an example output for a supplierof the retail store, consistent with the present disclosure.

FIG. 11C is a schematic illustration of an example output for a managerof the retail store, consistent with the present disclosure.

FIG. 11D is a schematic illustration of two examples outputs for anemployee of the retail store, consistent with the present disclosure.

FIG. 11E is a schematic illustration of an example output for an onlinecustomer of the retail store, consistent with the present disclosure.

FIG. 12 is a block diagram that illustrates some of the components of anelectronic visual display control system, consistent with the presentdisclosure.

FIG. 13A is a schematic cross-sectional side view illustration of anexemplary door for a retail storage container, consistent with thepresent disclosure.

FIG. 13B is a schematic cross-sectional side view illustration of anexemplary door for a retail storage container, consistent with thepresent disclosure.

FIG. 13C is a schematic cross-sectional view illustration of anexemplary door for a retail storage container, consistent with thepresent disclosure.

FIG. 14A-14F are schematic illustrations of exemplary retail storagecontainers, consistent with the present disclosure.

FIG. 15A-15H are schematic illustrations of exemplary retail storagecontainers, consistent with the present disclosure.

FIG. 16A-16F are schematic illustrations of exemplary retail shelves,consistent with the present disclosure.

FIG. 17 provides a flowchart of an exemplary method for controllinginformation displayed on an electronic visual display in a retail store,consistent with the present disclosure.

FIG. 18 provides a flowchart of an exemplary method for controllinginformation displayed on a transparent electronic visual display that ispart of a door for a retail storage container, consistent with thepresent disclosure.

FIG. 19 provides a flowchart of an exemplary method for selecting itemsfor presentation on electronic visual displays in retail stores,consistent with the present disclosure.

FIG. 20 provides a flowchart of an exemplary method for customizedpresentation of items on electronic visual displays in retail stores,consistent with the present disclosure.

FIG. 21 provides a flowchart of an exemplary method for selecting itemsfor presentation on electronic visual displays in retail stores,consistent with the present disclosure.

FIG. 22 provides a flowchart of an exemplary method for customizedpresentation of items on electronic visual displays in retail stores,consistent with the present disclosure.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions, or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples. Instead, the proper scope is defined by the appended claims.

The present disclosure is directed to systems and methods for processingimages captured in a retail store. As used herein, the term “retailstore” or simply “store” refers to an establishment offering productsfor sale by direct selection by customers physically or virtuallyshopping within the establishment. The retail store may be anestablishment operated by a single retailer (e.g., supermarket) or anestablishment that includes stores operated by multiple retailers (e.g.,a shopping mall). Embodiments of the present disclosure includereceiving an image depicting a store shelf having at least one productdisplayed thereon. As used herein, the term “store shelf” or simply“shelf” refers to any suitable physical structure which may be used fordisplaying products in a retail environment. In one embodiment the storeshelf may be part of a shelving unit including a number of individualstore shelves.

In another embodiment, the store shelf may include a display unit havinga single-level or multi-level surfaces.

Consistent with the present disclosure, the system may process imagesand image data acquired by a capturing device to determine informationassociated with products displayed in the retail store. The term“capturing device” refers to any device configured to acquire image datarepresentative of products displayed in the retail store. Examples ofcapturing devices may include a digital camera, a time-of-flight camera,a stereo camera, an active stereo camera, a depth camera, a Lidarsystem, a laser scanner, CCD based devices, or any other sensor basedsystem capable of converting received light into electric signals. Theterm “image data” refers to any form of data generated based on opticalsignals in the near-infrared, infrared, visible, and ultravioletspectrums (or any other suitable radiation frequency range). Consistentwith the present disclosure, the image data may include pixel datastreams, digital images, digital video streams, data derived fromcaptured images, and data that may be used to construct a 3D image. Theimage data acquired by a capturing device may be transmitted by wired orwireless transmission to a remote server. In one embodiment, thecapturing device may include a stationary camera with communicationlayers (e.g., a dedicated camera fixed to a store shelf, a securitycamera, and so forth). Such an embodiment is described in greater detailbelow with reference to FIG. 4A. In another embodiment, the capturingdevice may include a handheld device (e.g., a smartphone, a tablet, amobile station, a personal digital assistant, a laptop, and more) or awearable device (e.g., smart glasses, a smartwatch, a clip-on camera).Such an embodiment is described in greater detail below with referenceto FIG. 4B. In another embodiment, the capturing device may include arobotic device with one or more cameras operated remotely orautonomously (e.g., an autonomous robotic device, a drone, a robot on atrack, and more). Such an embodiment is described in greater detailbelow with reference to FIG. 4C.

In some embodiments, the capturing device may include one or more imagesensors. The term “image sensor” refers to a device capable of detectingand converting optical signals in the near-infrared, infrared, visible,and ultraviolet spectrums into electrical signals. The electricalsignals may be used to form image data (e.g., an image or a videostream) based on the detected signal. Examples of image sensors mayinclude semiconductor charge-coupled devices (CCD), active pixel sensorsin complementary metal-oxide-semiconductor (CMOS), or N-typemetal-oxide-semiconductors (NMOS, Live MOS). In some cases, the imagesensor may be part of a camera included in the capturing device.

Embodiments of the present disclosure further include analyzing imagesto detect and identify different products. As used herein, the term“detecting a product” may broadly refer to determining an existence ofthe product. For example, the system may determine the existence of aplurality of distinct products displayed on a store shelf. By detectingthe plurality of products, the system may acquire different detailsrelative to the plurality of products (e.g., how many products on astore shelf are associated with a same product type), but it does notnecessarily gain knowledge of the type of product. In contrast, the term“identifying a product” may refer to determining a unique identifierassociated with a specific type of product that allows inventorymanagers to uniquely refer to each product type in a product catalogue.Additionally or alternatively, the term “identifying a product” mayrefer to determining a unique identifier associated with a specificbrand of products that allows inventory managers to uniquely refer toproducts, e.g., based on a specific brand in a product catalogue.Additionally or alternatively, the term “identifying a product” mayrefer to determining a unique identifier associated with a specificcategory of products that allows inventory managers to uniquely refer toproducts, e.g., based on a specific category in a product catalogue. Insome embodiments, the identification may be made based at least in parton visual characteristics of the product (e.g., size, shape, logo, text,color, and so forth). The unique identifier may include any codes thatmay be used to search a catalog, such as a series of digits, letters,symbols, or any combinations of digits, letters, and symbols. Consistentwith the present disclosure, the terms “determining a type of a product”and “determining a product type” may also be used interchangeably inthis disclosure with reference to the term “identifying a product.”

Embodiments of the present disclosure further include determining atleast one characteristic of the product for determining the type of theproduct. As used herein, the term “characteristic of the product” refersto one or more visually discernable features attributed to the product.Consistent with the present disclosure, the characteristic of theproduct may assist in classifying and identifying the product. Forexample, the characteristic of the product may be associated with theornamental design of the product, the size of the product, the shape ofthe product, the colors of the product, the brand of the product, a logoor text associated with the product (e.g., on a product label), andmore. In addition, embodiments of the present disclosure further includedetermining a confidence level associated with the determined type ofthe product. The term “confidence level” refers to any indication,numeric or otherwise, of a level (e.g., within a predetermined range)indicative of an amount of confidence the system has that the determinedtype of the product is the actual type of the product. For example, theconfidence level may have a value between 1 and 10, alternatively, theconfidence level may be expressed as a percentage.

In some cases, the system may compare the confidence level to athreshold. The term “threshold” as used herein denotes a referencevalue, a level, a point, or a range of values, for which, when theconfidence level is above it (or below it depending on a particular usecase), the system may follow a first course of action and, when theconfidence level is below it (or above it depending on a particular usecase), the system may follow a second course of action. The value of thethreshold may be predetermined for each type of product or may bedynamically selected based on different considerations. In oneembodiment, when the confidence level associated with a certain productis below a threshold, the system may obtain contextual information toincrease the confidence level. As used herein, the term “contextualinformation” (or “context”) refers to any information having a direct orindirect relationship with a product displayed on a store shelf. In someembodiments, the system may retrieve different types of contextualinformation from captured image data and/or from other data sources. Insome cases, contextual information may include recognized types ofproducts adjacent to the product under examination. In other cases,contextual information may include text appearing on the product,especially where that text may be recognized (e.g., via OCR) andassociated with a particular meaning. Other examples of types ofcontextual information may include logos appearing on the product, alocation of the product in the retail store, a brand name of theproduct, a price of the product, product information collected frommultiple retail stores, product information retrieved from a catalogassociated with a retail store, etc.

Reference is now made to FIG. 1, which shows an example of a system 100for analyzing information collected from retail stores 105 (for example,retail store 105A, retail store 105B, and retail store 105C). In oneembodiment, system 100 may represent a computer-based system that mayinclude computer system components, desktop computers, workstations,tablets, handheld computing devices, memory devices, and/or internalnetwork(s) connecting the components. System 100 may include or beconnected to various network computing resources (e.g., servers,routers, switches, network connections, storage devices, etc.) necessaryto support the services provided by system 100. In one embodiment,system 100 may enable identification of products in retail stores 105based on analysis of captured images. In another embodiment, system 100may enable a supply of information based on analysis of captured imagesto a market research entity 110 and to different suppliers 115 of theidentified products in retail stores 105 (for example, supplier 115A,supplier 115B, and supplier 115C). In another embodiment, system 100 maycommunicate with a user 120 (sometimes referred to herein as a customer,but which may include individuals associated with a retail environmentother than customers, such as store employee, data collection agent,etc.) about different products in retail stores 105. In one example,system 100 may receive images of products captured by user 120. Inanother example, system 100 may provide to user 120 informationdetermined based on automatic machine analysis of images captured by oneor more capturing devices 125 associated with retail stores 105.

System 100 may also include an image processing unit 130 to execute theanalysis of images captured by the one or more capturing devices 125.Image processing unit 130 may include a server 135 operatively connectedto a database 140. Image processing unit 130 may include one or moreservers connected by a communication network, a cloud platform, and soforth. Consistent with the present disclosure, image processing unit 130may receive raw or processed data from capturing device 125 viarespective communication links, and provide information to differentsystem components using a network 150. Specifically, image processingunit 130 may use any suitable image analysis technique including, forexample, object recognition, object detection, image segmentation,feature extraction, optical character recognition (OCR), object-basedimage analysis, shape region techniques, edge detection techniques,pixel-based detection, artificial neural networks, convolutional neuralnetworks, etc. In addition, image processing unit 130 may useclassification algorithms to distinguish between the different productsin the retail store. In some embodiments, image processing unit 130 mayutilize suitably trained machine learning algorithms and models toperform the product identification. Network 150 may facilitatecommunications and data exchange between different system componentswhen these components are coupled to network 150 to enable output ofdata derived from the images captured by the one or more capturingdevices 125. In some examples, the types of outputs that imageprocessing unit 130 can generate may include identification of products,indicators of product quantity, indicators of planogram compliance,indicators of service-improvement events (e.g., a cleaning event, arestocking event, a rearrangement event, etc.), and various reportsindicative of the performances of retail stores 105. Additional examplesof the different outputs enabled by image processing unit 130 aredescribed below with reference to FIGS. 11A-11E and throughout thedisclosure.

Consistent with the present disclosure, network 150 may be any type ofnetwork (including infrastructure) that provides communications,exchanges information, and/or facilitates the exchange of informationbetween the components of system 100. For example, network 150 mayinclude or be part of the Internet, a Local Area Network, wirelessnetwork (e.g., a Wi-Fi/302.11 network), or other suitable connections.In other embodiments, one or more components of system 100 maycommunicate directly through dedicated communication links, such as, forexample, a telephone network, an extranet, an intranet, the Internet,satellite communications, off-line communications, wirelesscommunications, transponder communications, a local area network (LAN),a wide area network (WAN), a virtual private network (VPN), and soforth.

In one example configuration, server 135 may be a cloud server thatprocesses images received directly (or indirectly) from one or morecapturing device 125 and processes the images to detect and/or identifyat least some of the plurality of products in the image based on visualcharacteristics of the plurality of products. The term “cloud server”refers to a computer platform that provides services via a network, suchas the Internet. In this example configuration, server 135 may usevirtual machines that may not correspond to individual hardware. Forexample, computational and/or storage capabilities may be implemented byallocating appropriate portions of desirable computation/storage powerfrom a scalable repository, such as a data center or a distributedcomputing environment. In one example, server 135 may implement themethods described herein using customized hard-wired logic, one or moreApplication Specific Integrated Circuits (ASICs) or Field ProgrammableGate Arrays (FPGAs), firmware, and/or program logic which, incombination with the computer system, cause server 135 to be aspecial-purpose machine.

In another example configuration, server 135 may be part of a systemassociated with a retail store that communicates with capturing device125 using a wireless local area network (WLAN) and may provide similarfunctionality as a cloud server. In this example configuration, server135 may communicate with an associated cloud server (not shown) andcloud database (not shown). The communications between the store serverand the cloud server may be used in a quality enforcement process, forupgrading the recognition engine and the software from time to time, forextracting information from the store level to other data users, and soforth. Consistent with another embodiment, the communications betweenthe store server and the cloud server may be discontinuous (purposely orunintentional) and the store server may be configured to operateindependently from the cloud server. For example, the store server maybe configured to generate a record indicative of changes in productplacement that occurred when there was a limited connection (or noconnection) between the store server and the cloud server, and toforward the record to the cloud server once connection is reestablished.

As depicted in FIG. 1, server 135 may be coupled to one or more physicalor virtual storage devices such as database 140. Server 135 may accessdatabase 140 to detect and/or identify products. The detection may occurthrough analysis of features in the image using an algorithm and storeddata. The identification may occur through analysis of product featuresin the image according to stored product models. Consistent with thepresent embodiment, the term “product model” refers to any type ofalgorithm or stored product data that a processor may access or executeto enable the identification of a particular product associated with theproduct model. For example, the product model may include a descriptionof visual and contextual properties of the particular product (e.g., theshape, the size, the colors, the texture, the brand name, the price, thelogo, text appearing on the particular product, the shelf associatedwith the particular product, adjacent products in a planogram, thelocation within the retail store, and so forth). In some embodiments, asingle product model may be used by server 135 to identify more than onetype of products, such as, when two or more product models are used incombination to enable identification of a product. For example, in somecases, a first product model may be used by server 135 to identify aproduct category (such models may apply to multiple product types, e.g.,shampoo, soft drinks, etc.), and a second product model may be used byserver 135 to identify the product type, product identity, or othercharacteristics associated with a product. In some cases, such productmodels may be applied together (e.g., in series, in parallel, in acascade fashion, in a decision tree fashion, etc.) to reach a productidentification. In other embodiments, a single product model may be usedby server 135 to identify a particular product type (e.g., 6-pack of 16oz Coca-Cola Zero).

Database 140 may be included on a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, or other type ofstorage device or tangible or non-transitory computer-readable medium.Database 140 may also be part of server 135 or separate from server 135.When database 140 is not part of server 135, server 135 may exchangedata with database 140 via a communication link. Database 140 mayinclude one or more memory devices that store data and instructions usedto perform one or more features of the disclosed embodiments. In oneembodiment, database 140 may include any suitable databases, rangingfrom small databases hosted on a work station to large databasesdistributed among data centers. Database 140 may also include anycombination of one or more databases controlled by memory controllerdevices (e.g., server(s), etc.) or software. For example, database 140may include document management systems, Microsoft SQL databases,SharePoint databases, Oracle™ databases, Sybase™ databases, otherrelational databases, or non-relational databases, such as mongo andothers.

Consistent with the present disclosure, image processing unit 130 maycommunicate with output devices 145 to present information derived basedon processing of image data acquired by capturing devices 125. The term“output device” is intended to include all possible types of devicescapable of outputting information from server 135 to users or othercomputer systems (e.g., a display screen, a speaker, a desktop computer,a laptop computer, mobile device, tablet, a PDA, etc.), such as 145A,145B, 145C and 145D. In one embodiment each of the different systemcomponents (i.e., retail stores 105, market research entity 110,suppliers 115, and users 120) may be associated with an output device145, and each system component may be configured to present differentinformation on the output device 145. In one example, server 135 mayanalyze acquired images including representations of shelf spaces. Basedon this analysis, server 135 may compare shelf spaces associated withdifferent products, and output device 145A may present market researchentity 110 with information about the shelf spaces associated withdifferent products. The shelf spaces may also be compared with salesdata, expired products data, and more. Consistent with the presentdisclosure, market research entity 110 may be a part of (or may workwith) supplier 115. In another example, server 135 may determine productcompliance to a predetermined planogram, and output device 145B maypresent to supplier 115 information about the level of productcompliance at one or more retail stores 105 (for example in a specificretail store 105, in a group of retail stores 105 associated withsupplier 115, in all retail stores 105, and so forth). The predeterminedplanogram may be associated with contractual obligations and/or otherpreferences related to the retailer methodology for placement ofproducts on the store shelves. In another example, server 135 maydetermine that a specific store shelf has a type of fault in the productplacement, and output device 145C may present to a manager of retailstore 105 a user-notification that may include information about acorrect display location of a misplaced product, information about astore shelf associated with the misplaced product, information about atype of the misplaced product, and/or a visual depiction of themisplaced product. In another example, server 135 may identify whichproducts are available on the shelf and output device 145D may presentto user 120 an updated list of products.

The components and arrangements shown in FIG. 1 are not intended tolimit the disclosed embodiments, as the system components used toimplement the disclosed processes and features may vary. In oneembodiment, system 100 may include multiple servers 135, and each server135 may host a certain type of service. For example, a first server mayprocess images received from capturing devices 125 to identify at leastsome of the plurality of products in the image, and a second server maydetermine from the identified products in retail stores 105 compliancewith contractual obligations between retail stores 105 and suppliers115. In another embodiment, system 100 may include multiple servers 135,a first type of servers 135 that may process information from specificcapturing devices 125 (e.g., handheld devices of data collection agents)or from specific retail stores 105 (e.g., a server dedicated to aspecific retail store 105 may be placed in or near the store). System100 may further include a second type of servers 135 that collect andprocess information from the first type of servers 135.

FIG. 2 is a block diagram representative of an example configuration ofserver 135. In one embodiment, server 135 may include a bus 200 (or anyother communication mechanism) that interconnects subsystems andcomponents for transferring information within server 135. For example,bus 200 may interconnect a processing device 202, a memory interface204, a network interface 206, and a peripherals interface 208 connectedto an I/O system 210.

Processing device 202, shown in FIG. 2, may include at least oneprocessor configured to execute computer programs, applications,methods, processes, or other software to execute particular instructionsassociated with embodiments described in the present disclosure. Theterm “processing device” refers to any physical device having anelectric circuit that performs a logic operation. For example,processing device 202 may include one or more processors, integratedcircuits, microchips, microcontrollers, microprocessors, all or part ofa central processing unit (CPU), graphics processing unit (GPU), digitalsignal processor (DSP), field programmable gate array (FPGA), or othercircuits suitable for executing instructions or performing logicoperations. Processing device 202 may include at least one processorconfigured to perform functions of the disclosed methods such as amicroprocessor manufactured by Intel™, Nvidia™, manufactured by AMD™,and so forth. Processing device 202 may include a single core ormultiple core processors executing parallel processes simultaneously. Inone example, processing device 202 may be a single core processorconfigured with virtual processing technologies. Processing device 202may implement virtual machine technologies or other technologies toprovide the ability to execute, control, run, manipulate, store, etc.,multiple software processes, applications, programs, etc. In anotherexample, processing device 202 may include a multiple-core processorarrangement (e.g., dual, quad core, etc.) configured to provide parallelprocessing functionalities to allow a device associated with processingdevice 202 to execute multiple processes simultaneously. It isappreciated that other types of processor arrangements could beimplemented to provide the capabilities disclosed herein.

Consistent with the present disclosure, the methods and processesdisclosed herein may be performed by server 135 as a result ofprocessing device 202 executing one or more sequences of one or moreinstructions contained in a non-transitory computer-readable storagemedium. As used herein, a non-transitory computer-readable storagemedium refers to any type of physical memory on which information ordata readable by at least one processor can be stored. Examples includerandom access memory (RAM), read-only memory (ROM), volatile memory,nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, anyother optical data storage medium, any physical medium with patterns ofholes, a RAM, a PROM, an EPROM, a FLASH-EPROM or any other flash memory,NVRAM, a cache, a register, any other memory chip or cartridge, andnetworked versions of the same. The terms “memory” and“computer-readable storage medium” may refer to multiple structures,such as a plurality of memories or computer-readable storage mediumslocated within server 135, or at a remote location. Additionally, one ormore computer-readable storage mediums can be utilized in implementing acomputer-implemented method. The term “computer-readable storage medium”should be understood to include tangible items and exclude carrier wavesand transient signals.

According to one embodiment, server 135 may include network interface206 (which may also be any communications interface) coupled to bus 200.Network interface 206 may provide one-way or two-way data communicationto a local network, such as network 150. Network interface 206 mayinclude an integrated services digital network (ISDN) card, cable modem,satellite modem, or a modem to provide a data communication connectionto a corresponding type of telephone line. As another example, networkinterface 206 may include a local area network (LAN) card to provide adata communication connection to a compatible LAN. In anotherembodiment, network interface 206 may include an Ethernet port connectedto radio frequency receivers and transmitters and/or optical (e.g.,infrared) receivers and transmitters. The specific design andimplementation of network interface 206 depends on the communicationsnetwork(s) over which server 135 is intended to operate. As describedabove, server 135 may be a cloud server or a local server associatedwith retail store 105. In any such implementation, network interface 206may be configured to send and receive electrical, electromagnetic, oroptical signals, through wires or wirelessly, that may carry analog ordigital data streams representing various types of information. Inanother example, the implementation of network interface 206 may besimilar or identical to the implementation described below for networkinterface 306.

Server 135 may also include peripherals interface 208 coupled to bus200. Peripherals interface 208 may be connected to sensors, devices, andsubsystems to facilitate multiple functionalities. In one embodiment,peripherals interface 208 may be connected to I/O system 210 configuredto receive signals or input from devices and provide signals or outputto one or more devices that allow data to be received and/or transmittedby server 135. In one embodiment I/O system 210 may include or beassociated with output device 145. For example, I/O system 210 mayinclude a touch screen controller 212, an audio controller 214, and/orother input controller(s) 216. Touch screen controller 212 may becoupled to a touch screen 218. Touch screen 218 and touch screencontroller 212 can, for example, detect contact, movement, or breakthereof using any of a plurality of touch sensitivity technologies,including but not limited to capacitive, resistive, infrared, andsurface acoustic wave technologies as well as other proximity sensorarrays or other elements for determining one or more points of contactwith touch screen 218. Touch screen 218 may also, for example, be usedto implement virtual or soft buttons and/or a keyboard. In addition toor instead of touch screen 218, I/O system 210 may include a displayscreen (e.g., CRT, LCD, etc.), virtual reality device, augmented realitydevice, and so forth. Specifically, touch screen controller 212 (ordisplay screen controller) and touch screen 218 (or any of thealternatives mentioned above) may facilitate visual output from server135. Audio controller 214 may be coupled to a microphone 220 and aspeaker 222 to facilitate voice-enabled functions, such as voicerecognition, voice replication, digital recording, and telephonyfunctions. Specifically, audio controller 214 and speaker 222 mayfacilitate audio output from server 135. The other input controller(s)216 may be coupled to other input/control devices 224, such as one ormore buttons, keyboards, rocker switches, thumb-wheel, infrared port,USB port, image sensors, motion sensors, depth sensors, and/or a pointerdevice such as a computer mouse or a stylus.

In some embodiments, processing device 202 may use memory interface 204to access data and a software product stored on a memory device 226.Memory device 226 may include operating system programs for server 135that perform operating system functions when executed by the processingdevice. By way of example, the operating system programs may includeMicrosoft Windows™, Unix™, Linux™, Apple™ operating systems, personaldigital assistant (PDA) type operating systems such as Apple iOS, GoogleAndroid, Blackberry OS, or other types of operating systems.

Memory device 226 may also store communication instructions 228 tofacilitate communicating with one or more additional devices (e.g.,capturing device 125), one or more computers (e.g., output devices145A-145D) and/or one or more servers. Memory device 226 may includegraphical user interface instructions 230 to facilitate graphic userinterface processing; image processing instructions 232 to facilitateimage data processing-related processes and functions; sensor processinginstructions 234 to facilitate sensor-related processing and functions;web browsing instructions 236 to facilitate web browsing-relatedprocesses and functions; and other software instructions 238 tofacilitate other processes and functions. Each of the above identifiedinstructions and applications may correspond to a set of instructionsfor performing one or more functions described above. These instructionsneed not be implemented as separate software programs, procedures, ormodules. Memory device 226 may include additional instructions or fewerinstructions. Furthermore, various functions of server 135 may beimplemented in hardware and/or in software, including in one or moresignal processing and/or application specific integrated circuits. Forexample, server 135 may execute an image processing algorithm toidentify in received images one or more products and/or obstacles, suchas shopping carts, people, and more.

In one embodiment, memory device 226 may store database 140. Database140 may include product type model data 240 (e.g., an imagerepresentation, a list of features, a model obtained by training machinelearning algorithm using training examples, an artificial neuralnetwork, and more) that may be used to identify products in receivedimages; contract-related data 242 (e.g., planograms, promotions data,etc.) that may be used to determine if the placement of products on thestore shelves and/or the promotion execution are consistent withobligations of retail store 105; catalog data 244 (e.g., retail storechain's catalog, retail store's master file, etc.) that may be used tocheck if all product types that should be offered in retail store 105are in fact in the store, if the correct price is displayed next to anidentified product, etc.; inventory data 246 that may be used todetermine if additional products should be ordered from suppliers 115;employee data 248 (e.g., attendance data, records of training provided,evaluation and other performance-related communications, productivityinformation, etc.) that may be used to assign specific employees tocertain tasks; and calendar data 250 (e.g., holidays, national days,international events, etc.) that may be used to determine if a possiblechange in a product model is associated with a certain event. In otherembodiments of the disclosure, database 140 may store additional typesof data or fewer types of data. Furthermore, various types of data maybe stored in one or more memory devices other than memory device 226.

The components and arrangements shown in FIG. 2 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of server 135. For example, not all components may beessential for the operation of server 135 in all cases. Any componentmay be located in any appropriate part of server 135, and the componentsmay be rearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. For example, some serversmay not include some of the elements shown in I/O system 215.

FIG. 3 is a block diagram representation of an example configuration ofcapturing device 125. In one embodiment, capturing device 125 mayinclude a processing device 302, a memory interface 304, a networkinterface 306, and a peripherals interface 308 connected to image sensor310. These components can be separated or can be integrated in one ormore integrated circuits. The various components in capturing device 125can be coupled by one or more communication buses or signal lines (e.g.,bus 300). Different aspects of the functionalities of the variouscomponents in capturing device 125 may be understood from thedescription above regarding components of server 135 having similarfunctionality.

According to one embodiment, network interface 306 may be used tofacilitate communication with server 135. Network interface 306 may bean Ethernet port connected to radio frequency receivers and transmittersand/or optical receivers and transmitters. The specific design andimplementation of network interface 306 depends on the communicationsnetwork(s) over which capturing device 125 is intended to operate. Forexample, in some embodiments, capturing device 125 may include a networkinterface 306 designed to operate over a GSM network, a GPRS network, anEDGE network, a Wi-Fi or WiMax network, a Bluetooth® network, etc. Inanother example, the implementation of network interface 306 may besimilar or identical to the implementation described above for networkinterface 206.

In the example illustrated in FIG. 3, peripherals interface 308 ofcapturing device 125 may be connected to at least one image sensor 310associated with at least one lens 312 for capturing image data in anassociated field of view. In some configurations, capturing device 125may include a plurality of image sensors associated with a plurality oflenses 312. In other configurations, image sensor 310 may be part of acamera included in capturing device 125. According to some embodiments,peripherals interface 308 may also be connected to other sensors (notshown), such as a motion sensor, a light sensor, infrared sensor, soundsensor, a proximity sensor, a temperature sensor, a biometric sensor, orother sensing devices to facilitate related functionalities. Inaddition, a positioning sensor may also be integrated with, or connectedto, capturing device 125. For example, such positioning sensor may beimplemented using one of the following technologies: Global PositioningSystem (GPS), GLObal NAvigation Satellite System (GLONASS), Galileoglobal navigation system, BeiDou navigation system, other GlobalNavigation Satellite Systems (GNSS), Indian Regional NavigationSatellite System (IRNSS), Local Positioning Systems (LPS), Real-TimeLocation Systems (RTLS), Indoor Positioning System (IPS), Wi-Fi basedpositioning systems, cellular triangulation, and so forth. For example,the positioning sensor may be built into mobile capturing device 125,such as smartphone devices. In another example, position software mayallow mobile capturing devices to use internal or external positioningsensors (e.g., connecting via a serial port or Bluetooth).

Consistent with the present disclosure, capturing device 125 may includedigital components that collect data from image sensor 310, transform itinto an image, and store the image on a memory device 314 and/ortransmit the image using network interface 306. In one embodiment,capturing device 125 may be fixedly mountable to a store shelf or toother objects in the retail store (such as walls, ceilings, floors,refrigerators, checkout stations, displays, dispensers, rods which maybe connected to other objects in the retail store, and so forth). In oneembodiment, capturing device 125 may be split into at least two housingssuch that only image sensor 310 and lens 312 may be visible on the storeshelf, and the rest of the digital components may be located in aseparate housing. An example of this type of capturing device isdescribed below with reference to FIGS. 5-7.

Consistent with the present disclosure, capturing device 125 may usememory interface 304 to access memory device 314. Memory device 314 mayinclude high-speed, random access memory and/or non-volatile memory suchas one or more magnetic disk storage devices, one or more opticalstorage devices, and/or flash memory (e.g., NAND, NOR) to store capturedimage data. Memory device 314 may store operating system instructions316, such as DARWIN, RTXC, LINUX, iOS, UNIX, LINUX, OS X, WINDOWS, or anembedded operating system such as VXWorkS. Operating system 316 caninclude instructions for handling basic system services and forperforming hardware dependent tasks. In some implementations, operatingsystem 316 may include a kernel (e.g., UNIX kernel, LINUX kernel, and soforth). In addition, memory device 314 may store capturing instructions318 to facilitate processes and functions related to image sensor 310;graphical user interface instructions 320 that enables a user associatedwith capturing device 125 to control the capturing device and/or toacquire images of an area-of-interest in a retail establishment; andapplication instructions 322 to facilitate a process for monitoringcompliance of product placement or other processes.

The components and arrangements shown in FIG. 3 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of capturing device 125. For example, not all componentsare essential for the operation of capturing device 125 in all cases.Any component may be located in any appropriate part of capturing device125, and the components may be rearranged into a variety ofconfigurations while providing the functionality of the disclosedembodiments. For example, some capturing devices may not have lenses,and other capturing devices may include an external memory deviceinstead of memory device 314.

FIGS. 4A-4C illustrate example configurations for capturing image datain retail store 105 according to disclosed embodiments. FIG. 4Aillustrates how an aisle 400 of retail store 105 may be imaged using aplurality of capturing devices 125 fixedly connected to store shelves.FIG. 4B illustrates how aisle 400 of retail store 105 may be imagedusing a handheld communication device. FIG. 4C illustrates how aisle 400of retail store 105 may be imaged by robotic devices equipped withcameras.

With reference to FIG. 4A and consistent with the present disclosure,retail store 105 may include a plurality of capturing devices 125fixedly mounted (for example, to store shelves, walls, ceilings, floors,refrigerators, checkout stations, displays, dispensers, rods which maybe connected to other objects in the retail store, and so forth) andconfigured to collect image data. As depicted, one side of an aisle 400may include a plurality of capturing devices 125 (e.g., 125A, 125B, and125C) fixedly mounted thereon and directed such that they may captureimages of an opposing side of aisle 400. The plurality of capturingdevices 125 may be connected to an associated mobile power source (e.g.,one or more batteries), to an external power supply (e.g., a powergrid), obtain electrical power from a wireless power transmissionsystem, and so forth. As depicted in FIG. 4A, the plurality of capturingdevices 125 may be placed at different heights and at least theirvertical fields of view may be adjustable. Generally, both sides ofaisle 400 may include capturing devices 125 in order to cover both sidesof aisle 400.

Differing numbers of capturing devices 125 may be used to cover shelvingunit 402. In addition, there may be an overlap region in the horizontalfield of views of some of capturing devices 125. For example, thehorizontal fields of view of capturing devices (e.g., adjacent capturingdevices) may at least partially overlap with one another. In anotherexample, one capturing device may have a lower field of view than thefield of view of a second capturing device, and the two capturingdevices may have at least partially overlapping fields of view.According to one embodiment, each capturing device 125 may be equippedwith network interface 306 for communicating with server 135. In oneembodiment, the plurality of capturing devices 125 in retail store 105may be connected to server 135 via a single WLAN. Network interface 306may transmit information associated with a plurality of images capturedby the plurality of capturing devices 125 for analysis purposes. In oneexample, server 135 may determine an existence of an occlusion event(such as, by a person, by store equipment, such as a ladder, cart, etc.)and may provide a notification to resolve the occlusion event. Inanother example, server 135 may determine if a disparity exists betweenat least one contractual obligation and product placement as determinedbased on automatic analysis of the plurality of images. The transmittedinformation may include raw images, cropped images, processed imagedata, data about products identified in the images, and so forth.Network interface 306 may also transmit information identifying thelocation of the plurality capturing devices 125 in retail store 105.

With reference to FIG. 4B and consistent with the present disclosure,server 135 may receive image data captured by users 120. In a firstembodiment, server 135 may receive image data acquired by storeemployees. In one implementation, a handheld device of a store employee(e.g., capturing device 125D) may display a real-time video streamcaptured by the image sensor of the handheld device. The real-time videostream may be augmented with markings identifying to the store employeean area-of-interest that needs manual capturing of images. One of thesituations in which manual image capture may be desirable may occurwhere the area-of-interest is outside the fields of view of a pluralityof cameras fixedly connected to store shelves in aisle 400. In othersituations, manual capturing of images of an area-of-interest may bedesirable when a current set of acquired images is out of date (e.g.,obsolete in at least one respect) or of poor quality (e.g., lackingfocus, obstacles, lesser resolution, lack of light, and so forth).Additional details of this embodiment are described in Applicant'sInternational Patent Application No. PCT/IB2018/001107, which isincorporated herein by reference.

In a second embodiment, server 135 may receive image data acquired bycrowd sourcing. In one exemplary implementation, server 135 may providea request to a detected mobile device for an updated image of thearea-of-interest in aisle 400. The request may include an incentive(e.g., $2 discount) to user 120 for acquiring the image. In response tothe request, user 120 may acquire and transmit an up-to-date image ofthe area-of-interest. After receiving the image from user 120, server135 may transmit the accepted incentive or agreed upon reward to user120. The incentive may comprise a text notification and a redeemablecoupon. In some embodiments, the incentive may include a redeemablecoupon for a product associated with the area-of-interest. Server 135may generate image-related data based on aggregation of data from imagesreceived from crowd sourcing and from images received from a pluralityof cameras fixedly connected to store shelves. Additional details ofthis embodiment are described in Applicant's International PatentApplication No. PCT/IB2017/000919, which is incorporated herein byreference.

With reference to FIG. 4C and consistent with the present disclosure,server 135 may receive image data captured by robotic devices withcameras traversing in aisle 400. The present disclosure is not limitedto the type of robotic devices used to capture images of retail store105. In some embodiments, the robotic devices may include a robot on atrack (e.g., a Cartesian robot configured to move along an edge of ashelf or in parallel to a shelf, such as capturing device 125E), a drone(e.g., capturing device 125F), and/or a robot that may move on the floorof the retail store (e.g., a wheeled robot such as capturing device125G, a legged robot, a snake-like robot, and so forth). The roboticdevices may be controlled by server 135 and may be operated remotely orautonomously. In one example, server 135 may instruct capturing device125E to perform periodic scans at times when no customers or otherobstructions are identified in aisle 400. Specifically, capturing device125E may be configured to move along store shelf 404 and to captureimages of products placed on store shelf 404, products placed on storeshelf 406, or products located on shelves opposite store shelf (e.g.,store shelf 408). In another example, server 135 may instruct capturingdevice 125F to perform a scan of all the area of retail store 105 beforethe opening hour. In another example, server 135 may instruct capturingdevice 125G to capture a specific area-of-interest, similar as describedabove with reference to receiving images acquired by the storeemployees. In some embodiments, robotic capturing devices (such as 125Fand 125G) may include an internal processing unit that may allow them tonavigate autonomously within retail store 105. For example, the roboticcapturing devices may use input from sensors (e.g., image sensors, depthsensors, proximity sensors, etc.), to avoid collision with objects orpeople, and to complete the scan of the desired area of retail store105.

As discussed above with reference to FIG. 4A, the image datarepresentative of products displayed on store shelves may be acquired bya plurality of stationary capturing devices 125 fixedly mounted in theretail store. One advantage of having stationary image capturing devicesspread throughout retail store 105 is the potential for acquiringproduct images from set locations and on an ongoing basis such thatup-to-date product status may be determined for products throughout aretail store at any desired periodicity (e.g., in contrast to a movingcamera system that may acquire product images more infrequently).However, there may be certain challenges in this approach. The distancesand angles of the image capturing devices relative to the capturedproducts should be selected such as to enable adequate productidentification, especially when considered in view of image sensorresolution and/or optics specifications. For example, a capturing deviceplaced on the ceiling of retail store 105 may have sufficientresolutions and optics to enable identification of large products (e.g.,a pack of toilet paper), but may be insufficient for identifying smallerproducts (e.g., deodorant packages). The image capturing devices shouldnot occupy shelf space that is reserved for products for sale. The imagecapturing devices should not be positioned in places where there is alikelihood that their fields of view will be regularly blocked bydifferent objects. The image capturing devices should be able tofunction for long periods of time with minimum maintenance. For example,a requirement for frequent replacement of batteries may render certainimage acquisition systems cumbersome to use, especially where many imageacquisition devices are in use throughout multiple locations in a retailstore and across multiple retail stores. The image capturing devicesshould also include processing capabilities and transmissioncapabilities for providing real time or near real time image data aboutproducts. The disclosed image acquisition systems address thesechallenges.

FIG. 5A illustrates an example of a system 500 for acquiring images ofproducts in retail store 105. Throughout the disclosure, capturingdevice 125 may refer to a system, such as system 500 shown in FIG. 5A.As shown, system 500 may include a first housing 502 configured forlocation on a retail shelving unit (e.g., as illustrated in FIG. 5B),and a second housing 504 configured for location on the retail shelvingunit separate from first housing 502. The first and the second housingmay be configured for mounting on the retail shelving unit in anysuitable way (e.g., screws, bolts, clamps, adhesives, magnets,mechanical means, chemical means, and so forth). In some embodiments,first housing 502 may include an image capture device 506 (e.g., acamera module that may include image sensor 310) and second housing 504may include at least one processor (e.g., processing device 302)configured to control image capture device 506 and also to control anetwork interface (e.g., network interface 306) for communicating with aremote server (e.g., server 135).

System 500 may also include a data conduit 508 extending between firsthousing 502 and second housing 504. Data conduit 508 may be configuredto enable transfer of control signals from the at least one processor toimage capture device 506 and to enable collection of image data acquiredby image capture device 506 for transmission by the network interface.Consistent with the present disclosure, the term “data conduit” mayrefer to a communications channel that may include either a physicaltransmission medium such as a wire or a logical connection over amultiplexed medium such as a radio channel In some embodiments, dataconduit 508 may be used for conveying image data from image capturedevice 506 to at least one processor located in second housing 504.Consistent with one implementation of system 500, data conduit 508 mayinclude flexible printed circuits and may have a length of at leastabout 5 cm, at least about 10 cm, at least about 15 cm, etc. The lengthof data conduit 508 may be adjustable to enable placement of firsthousing 502 separately from second housing 504. For example, in someembodiments, data conduit may be retractable within second housing 504such that the length of data conduit exposed between first housing 502and second housing 504 may be selectively adjusted.

In one embodiment, the length of data conduit 508 may enable firsthousing 502 to be mounted on a first side of a horizontal store shelffacing the aisle (e.g., store shelf 510 illustrated in FIG. 5B) andsecond housing 504 to be mounted on a second side of store shelf 510that faces the direction of the ground (e.g., an underside of a storeshelf). In this embodiment, data conduit 508 may be configured to bendaround an edge of store shelf 510 or otherwise adhere/follow contours ofthe shelving unit. For example, a first portion of data conduit 508 maybe configured for location on the first side of store shelf 510 (e.g., aside facing an opposing retail shelving unit across an aisle) and asecond portion of data conduit 508 may be configured for location on asecond side of store shelf 510 (e.g., an underside of the shelf, whichin some cases may be orthogonal to the first side). The second portionof data conduit 508 may be longer than the first portion of data conduit508. Consistent with another embodiment, data conduit 508 may beconfigured for location within an envelope of a store shelf. Forexample, the envelope may include the outer boundaries of a channellocated within a store shelf, a region on an underside of an

L-shaped store shelf, a region between two store shelves, etc.Consistent with another implementation of system 500 discussed below,data conduit 508 may include a virtual conduit associated with awireless communications link between first housing 502 and secondhousing 504.

FIG. 5B illustrates an exemplary configuration for mounting firsthousing 502 on store shelf 510. Consistent with the present disclosure,first housing 502 may be placed on store shelf 510, next to or embeddedin a plastic cover that may be used for displaying prices.Alternatively, first housing 502 may be placed or mounted on any otherlocation in retail store 105. For example, first housing 502 may beplaced or mounted on the walls, on the ceiling, on refrigerator units,on display units, and more. The location and/or orientation of firsthousing 502 may be selected such that a field of view of image capturedevice 506 may cover at least a portion of an opposing retail shelvingunit. Consistent with the present disclosure, image capture device 506may have a view angle of between 50 and 80 degrees, about 62 degrees,about 67 degrees, or about 75 degrees. Consistent with the presentdisclosure, image capture device 506 may include an image sensor havingsufficient image resolution to enable detection of text associated withlabels on an opposing retail shelving unit. In one embodiment, the imagesensor may include m*n pixels. For example, image capture device 506 mayhave an 8 MP image sensor that includes an array of 3280*2464 pixels.Each pixel may include at least one photo-voltaic cell that converts thephotons of the incident light to an electric signal. The electricalsignal may be converted to digital data by an A/D converter andprocessed by the image processor (ISP). In one embodiment, the imagesensor of image capture device 506 may be associated with a pixel sizeof between 1.1×1.1 um2 and 1.7×1.7 um2, for example, 1.4×1.4 um2.

Consistent with the present disclosure, image capture device 506 may beassociated with a lens (e.g., lens 312) having a fixed focal lengthselected according to a distance expected to be encountered betweenretail shelving units on opposite sides of an aisle (e.g., distance d1shown in FIG. 6A) and/or according to a distance expected to beencountered between a side of a shelving unit facing the aisle on oneside of an aisle and a side of a shelving unit facing away of the aisleon the other side of the aisle (e.g., distance d2 shown in FIG. 6A). Thefocal length may also be based on any other expected distance betweenthe image acquisition device and products to be imaged. As used herein,the term “focal length” refers to the distance from the optical centerof the lens to a point where objects located at the point aresubstantially brought into focus. In contrast to zoom lenses, in fixedlenses the focus is not adjustable. The focus is typically set at thetime of lens design and remains fixed. In one embodiment, the focallength of lens 312 may be selected based on the distance between twosides of aisles in the retail store (e.g., distance d1, distance d2, andso forth). In some embodiments, image capture device 506 may include alens with a fixed focal length having a fixed value between 2.5 mm and4.5 mm, such as about 3.1 mm, about 3.4 mm, about 3.7 mm. For example,when distance d1 between two opposing retail shelving units is about 2meters, the focal length of the lens may be about 3.6 mm. Unlessindicated otherwise, the term “about” with regards to a numeric value isdefined as a variance of up to 5% with respect to the stated value. Ofcourse, image capture devices having non-fixed focal lengths may also beused depending on the requirements of certain imaging environments, thepower and space resources available, etc.

FIG. 5C illustrates an exploded view of second housing 504. In someembodiments, the network interface located in second housing 504 (e.g.,network interface 306) may be configured to transmit to server 135information associated with a plurality of images captured by imagecapture device 506. For example, the transmitted information may be usedto determine if a disparity exists between at least one contractualobligation (e.g. planogram) and product placement. In one example, thenetwork interface may support transmission speeds of 0.5 Mb/s, 1 Mb/s, 5Mb/s, or more. Consistent with the present disclosure, the networkinterface may allow different modes of operations to be selected, suchas: high-speed, slope-control, or standby. In high-speed mode,associated output drivers may have fast output rise and fall times tosupport high-speed bus rates; in slope-control, the electromagneticinterference may be reduced and the slope (i.e., the change of voltageper unit of time) may be proportional to the current output; and instandby mode, the transmitter may be switched off and the receiver mayoperate at a lower current.

Consistent with the present disclosure, second housing 504 may include apower port 512 for conveying energy from a power source to first housing502. In one embodiment, second housing 504 may include a section for atleast one mobile power source 514 (e.g., in the depicted configurationthe section is configured to house four batteries). The at least onemobile power source may provide sufficient power to enable image capturedevice 506 to acquire more than 1,000 pictures, more than 5,000pictures, more than 10,000 pictures, or more than 15,000 pictures, andto transmit them to server 135. In one embodiment, mobile power source514 located in a single second housing 504 may power two or more imagecapture devices 506 mounted on the store shelf. For example, as depictedin FIGS. 6A and 6B, a single second housing 504 may be connected to aplurality of first housings 502 with a plurality of image capturedevices 506 covering different (overlapping or non-overlapping) fieldsof view. Accordingly, the two or more image capture devices 506 may bepowered by a single mobile power source 514 and/or the data captured bytwo or more image capture devices 506 may be processed to generate apanoramic image by a single processing device located in second housing504. In addition to mobile power source 514 or as an alternative tomobile power source 514, second housing 504 may also be connected to anexternal power source. For example, second housing 504 may be mounted toa store shelf and connected to an electric power grid. In this example,power port 512 may be connected to the store shelf through a wire forproviding electrical power to image capture device 506. In anotherexample, a retail shelving unit or retail store 105 may include awireless power transmission system, and power port 512 may be connectedto a device configured to obtain electrical power from the wirelesspower transmission system. In addition, as discussed below, system 500may use power management policies to reduce the power consumption. Forexample, system 500 may use selective image capturing and/or selectivetransmission of images to reduce the power consumption or conservepower.

FIG. 6A illustrates a schematic diagram of a top view of aisle 600 inretail store 105 with multiple image acquisition systems 500 (e.g.,500A, 500B, 500C, 500D, and 500E) deployed thereon for acquiring imagesof products. Aisle 600 may include a first retail shelving unit 602 anda second retail shelving unit 604 that opposes first retail shelvingunit 602. In some embodiments, different numbers of systems 500 may bemounted on opposing retail shelving units. For example, system 500A(including first housing 502A, second housing 504A, and data conduit508A), system 500B (including first housing 502B second housing 504B,and data conduit 508B), and system 500C (including first housing 502C,second housing 504C, and data conduit 508C) may be mounted on firstretail shelving unit 602; and system 500D (including first housing502D1, first housing 502D2, second housing 504D, and data conduits 508D1and 508D2) and system 500E (including first housing 502E1, first housing502E2, second housing 504E, and data conduits 508E1 and 508E2) may bemounted on second retail shelving unit 604. Consistent with the presentdisclosure, image capture device 506 may be configured relative to firsthousing 502 such that an optical axis of image capture device 506 isdirected toward an opposing retail shelving unit when first housing 502is fixedly mounted on a retail shelving unit. For example, optical axis606 of the image capture device associated with first housing 502B maybe directed towards second retail shelving unit 604 when first housing502B is fixedly mounted on first retail shelving unit 602. A singleretail shelving unit may hold a number of systems 500 that include aplurality of image capturing devices. Each of the image capturingdevices may be associated with a different field of view directed towardthe opposing retail shelving unit. Different vantage points ofdifferently located image capture devices may enable image acquisitionrelative to different sections of a retail shelf. For example, at leastsome of the plurality of image capturing devices may be fixedly mountedon shelves at different heights. Examples of such a deployment areillustrated in FIGS. 4A and 6B.

As shown in FIG. 6A each first housing 502 may be associated with a dataconduit 508 that enables exchanging of information (e.g., image data,control signals, etc.) between the at least one processor located insecond housing 504 and image capture device 506 located in first housing502. In some embodiments, data conduit 508 may include a wiredconnection that supports data-transfer and may be used to power imagecapture device 506 (e.g., data conduit 508A, data conduit 508B, dataconduit 508D1, data conduit 508D2, data conduit 508E1, and data conduit508E2). Consistent with these embodiments, data conduit 508 may complywith a wired standard such as USB, Micro-USB, HDMI, Micro-HDMI,Firewire, Apple, etc. In other embodiments, data conduit 508 may be awireless connection, such as a dedicated communications channel betweenthe at least one processor located in second housing 504 and imagecapture device 506 located in first housing 502 (e.g., data conduit508C). In one example, the communications channel may be established bytwo Near Field Communication (NFC) transceivers. In other examples,first housing 502 and second housing 504 may include interface circuitsthat comply with other short-range wireless standards such as Bluetooth,WiFi, ZigBee, etc.

In some embodiments of the disclosure, the at least one processor ofsystem 500 may cause at least one image capture device 506 toperiodically capture images of products located on an opposing retailshelving unit (e.g., images of products located on a shelf across anaisle from the shelf on which first housing 502 is mounted). The term“periodically capturing images” includes capturing an image or images atpredetermined time intervals (e.g., every minute, every 30 minutes,every 150 minutes, every 300 minutes, etc.), capturing video, capturingan image every time a status request is received, and/or capturing animage subsequent to receiving input from an additional sensor, forexample, an associated proximity sensor. Images may also be capturedbased on various other triggers or in response to various other detectedevents. In some embodiments, system 500 may receive an output signalfrom at least one sensor located on an opposing retail shelving unit.For example, system 500B may receive output signals from a sensingsystem located on second retail shelving unit 604. The output signalsmay be indicative of a sensed lifting of a product from second retailshelving unit 604 or a sensed positioning of a product on second retailshelving unit 604. In response to receiving the output signal from theat least one sensor located on second retail shelving unit 604, system500B may cause image capture device 506 to capture one or more images ofsecond retail shelving unit 604. Additional details on a sensing system,including the at least one sensor that generates output signalsindicative of a sensed lifting of a product from an opposing retailshelving unit, is discussed below with reference to FIGS. 8-10.

Consistent with embodiments of the disclosure, system 500 may detect anobject 608 in a selected area between first retail shelving unit 602 andsecond retail shelving unit 604. Such detection may be based on theoutput of one or more dedicated sensors (e.g., motion detectors, etc.)and/or may be based on image analysis of one or more images acquired byan image acquisition device. Such images, for example, may include arepresentation of a person or other object recognizable through variousimage analysis techniques (e.g., trained neural networks, Fouriertransform analysis, edge detection, filters, face recognition, and soforth). The selected area may be associated with distance d1 betweenfirst retail shelving unit 602 and second retail shelving unit 604. Theselected area may be within the field of view of image capture device506 or an area where the object causes an occlusion of a region ofinterest (such as a shelf, a portion of a shelf being monitored, andmore). Upon detecting object 608, system 500 may cause image capturedevice 506 to forgo image acquisition while object 608 is within theselected area. In one example, object 608 may be an individual, such asa customer or a store employee. In another example, detected object 608may be an inanimate object, such as a cart, box, carton, one or moreproducts, cleaning robots, etc. In the example illustrated in FIG. 6A,system 500A may detect that object 608 has entered into its associatedfield of view (e.g., using a proximity sensor) and may instruct imagecapturing device 506 to forgo image acquisition. In alternativeembodiments, system 500 may analyze a plurality of images acquired byimage capture device 506 and identify at least one image of theplurality of images that includes a representation of object 608.Thereafter, system 500 may avoid transmission of at least part of the atleast one identified image and/or information based on the at least oneidentified image to server 135.

As shown in FIG. 6A, the at least one processor contained in a secondhousing 504 may control a plurality of image capture devices 506contained in a plurality of first housings 502 (e.g., systems 500D and500E). Controlling image capturing device 506 may include instructingimage capturing device 506 to capture an image and/or transmit capturedimages to a remote server (e.g., server 135). In some cases, each of theplurality of image capture devices 506 may have a field of view that atleast partially overlaps with a field of view of at least one otherimage capture device 506 from among plurality of image capture devices506. In one embodiment, the plurality of image capture devices 506 maybe configured for location on one or more horizontal shelves and may bedirected to substantially different areas of the opposing first retailshelving unit. In this embodiment, the at least one processor maycontrol the plurality of image capture devices such that each of theplurality of image capture devices may capture an image at a differenttime. For example, system 500E may have a second housing 504E with atleast one processor that may instruct a first image capturing devicecontained in first housing 502E1 to capture an image at a first time andmay instruct a second image capturing device contained in first housing502E2 to capture an image at a second time which differs from the firsttime. Capturing images in different times (or forwarding them to the atleast one processor at different times) may assist in processing theimages and writing the images in the memory associated with the at leastone processor.

FIG. 6B illustrates a perspective view assembly diagram depicting aportion of a retail shelving unit 620 with multiple systems 500 (e.g.,500F, 500G, 500H, 500I, and 500J) deployed thereon for acquiring imagesof products. Retail shelving unit 620 may include horizontal shelves atdifferent heights. For example, horizontal shelves 622A, 622B, and 622Care located below horizontal shelves 622D, 622E, and 622F. In someembodiments, a different number of systems 500 may be mounted on shelvesat different heights. For example, system 500F (including first housing502F and second housing 504F), system 500G (including first housing 502Gand second housing 504G), and system 500H (including first housing 502Hand second housing 504H) may be mounted on horizontal shelves associatedwith a first height; and system 500I (including first housing 502I,second housing 5041, and a projector 632) and system 500J (includingfirst housing 502J1, first housing 502J2, and second housing 504J) maybe mounted on horizontal shelves associated with a second height. Insome embodiments, retail shelving unit 620 may include a horizontalshelf with at least one designated place (not shown) for mounting ahousing of image capturing device 506. The at least one designated placemay be associated with connectors such that first housing 502 may befixedly mounted on a side of horizontal shelf 622 facing an opposingretail shelving unit using the connectors.

Consistent with the present disclosure, system 500 may be mounted on aretail shelving unit that includes at least two adjacent horizontalshelves (e.g., shelves 622A and 622B) forming a substantially continuoussurface for product placement. The store shelves may include standardstore shelves or customized store shelves. A length of each store shelf622 may be at least 50 cm, less than 200 cm, or between 75 cm to 175 cm.In one embodiment, first housing 502 may be fixedly mounted on theretail shelving unit in a slit between two adjacent horizontal shelves.For example, first housing 502G may be fixedly mounted on retailshelving unit 620 in a slit between horizontal shelf 622B and horizontalshelf 622C. In another embodiment, first housing 502 may be fixedlymounted on a first shelf and second housing 504 may be fixedly mountedon a second shelf. For example, first housing 502I may be mounted onhorizontal shelf 622D and second housing 504I may be mounted onhorizontal shelf 622E. In another embodiment, first housing 502 may befixedly mounted on a retail shelving unit on a first side of ahorizontal shelf facing the opposing retail shelving unit and secondhousing 504 may be fixedly mounted on retail shelving unit 620 on asecond side of the horizontal shelf orthogonal to the first side. Forexample, first housing 502H may mounted on a first side 624 ofhorizontal shelf 622C next to a label and second housing 504H may bemounted on a second side 626 of horizontal shelf 622C that faces down(e.g., towards the ground or towards a lower shelf). In anotherembodiment, second housing 504 may be mounted closer to the back of thehorizontal shelf than to the front of the horizontal shelf. For example,second housing 504H may be fixedly mounted on horizontal shelf 622C onsecond side 626 closer to third side 628 of the horizontal shelf 622Cthan to first side 624. Third side 628 may be parallel to first side624. As mentioned above, data conduit 508 (e.g., data conduit 508H) mayhave an adjustable or selectable length for extending between firsthousing 502 and second housing 504. In one embodiment, when firsthousing 502H is fixedly mounted on first side 624, the length of dataconduit 508H may enable second housing 604H to be fixedly mounted onsecond side 626 closer to third side 628 than to first side 624.

As mentioned above, at least one processor contained in a single secondhousing 504 may control a plurality of image capture devices 506contained in a plurality of first housings 502 (e.g., system 500J). Insome embodiments, the plurality of image capture devices 506 may beconfigured for location on a single horizontal shelf and may be directedto substantially the same area of the opposing first retail shelvingunit (e.g., system 500D in FIG. 6A). In these embodiments, the imagedata acquired by the first image capture device and the second imagecapture device may enable a calculation of depth information (e.g.,based on image parallax information) associated with at least oneproduct positioned on an opposing retail shelving unit. For example,system 500J may have single second housing 504J with at least oneprocessor that may control a first image capturing device contained infirst housing 502J1 and a second image capturing device contained infirst housing 502J2. The distance d3 between the first image capturedevice contained in first housing 502J1 and the second image capturedevice contained in first housing 502J2 may be selected based on thedistance between retail shelving unit 620 and the opposing retailshelving unit (e g , similar to d1 and/or d2). For example, distance d3may be at least 5 cm, at least 10 cm, at least 15 cm, less than 40 cm,less than 30 cm, between about 5 cm to about 20 cm, or between about 10cm to about 15 cm. In another example, d3 may be a function of d1 and/ord2, a linear function of d1 and/or d2, a function of d1*log(d1) and/ord2*log(d2) such as a1*d1*log(d1) for some constant a1, and so forth. Thedata from the first image capturing device contained in first housing502J1 and the second image capturing device contained in first housing502J2 may be used to estimate the number of products on a store shelf ofretail shelving unit 602. In related embodiments, system 500 may controla projector (e.g., projector 632) and image capture device 506 that areconfigured for location on a single store shelf or on two separate storeshelves. For example, projector 632 may be mounted on horizontal shelf622E and image capture device 506I may be mounted on horizontal shelf622D. The image data acquired by image capture device 506 (e.g.,included in first housing 502I) may include reflections of lightpatterns projected from projector 632 on the at least one product and/orthe opposing retail shelving unit and may enable a calculation of depthinformation associated with at least one product positioned on theopposing retail shelving unit. The distance between projector 632 andthe image capture device contained in first housing 502I may be selectedbased on the distance between retail shelving unit 620 and the opposingretail shelving unit (e.g., similar to d1 and/or d2). For example, thedistance between the projector and the image capture device may be atleast 5 cm, at least 10 cm, at least 15 cm, less than 40 cm, less than30 cm, between about 5 cm to about 20 cm, or between about 10 cm toabout 15 cm. In another example, the distance between the projector andthe image capture device may be a function of d1 and/or d2, a linearfunction of d1 and/or d2, a function of d1*log(d1) and/or d2*log(d2)such as a1* d1*log(d1) for some constant a1, and so forth.

Consistent with the present disclosure, a central communication device630 may be located in retail store 105 and may be configured tocommunicate with server 135 (e.g., via an Internet connection). Thecentral communication device may also communicate with a plurality ofsystems 500 (for example, less than ten, ten, eleven, twelve, more thantwelve, and so forth). In some cases, at least one system of theplurality of systems 500 may be located in proximity to centralcommunication device 630. In the illustrated example, system 500F may belocated in proximity to central communication device 630. In someembodiments, at least some of systems 500 may communicate directly withat least one other system 500. The communications between some of theplurality of systems 500 may happen via a wired connection, such as thecommunications between system 500J and system 500I and thecommunications between system 500H and system 500G. Additionally oralternatively, the communications between some of the plurality ofsystems 500 may occur via a wireless connection, such as thecommunications between system 500G and system 500F and thecommunications between system 500I and system 500F. In some examples, atleast one system 500 may be configured to transmit captured image data(or information derived from the captured image data) to centralcommunication device 630 via at least two mediating systems 500, atleast three mediating systems 500, at least four mediating systems 500,or more. For example, system 500J may convey captured image data tocentral communication device 630 via system 500I and system 500F.

Consistent with the present disclosure, two (or more) systems 500 mayshare information to improve image acquisition. For example, system 500Jmay be configured to receive from a neighboring system 500I informationassociated with an event that system 500I had identified, and controlimage capture device 506 based on the received information. For example,system 500J may forgo image acquisition based on an indication fromsystem 500I that an object has entered or is about to enter its field ofview. Systems 500I and 500J may have overlapping fields of view ornon-overlapping fields of view. In addition, system 500J may alsoreceive (from system 500I) information that originates from centralcommunication device 630 and control image capture device 506 based onthe received information. For example, system 500I may receiveinstructions from central communication device 630 to capture an imagewhen suppler 115 inquiries about a specific product that is placed in aretail unit opposing system 500I. In some embodiments, a plurality ofsystems 500 may communicate with central communication device 630. Inorder to reduce or avoid network congestion, each system 500 mayidentify an available transmission time slot. Thereafter, each system500 may determine a default time slot for future transmissions based onthe identified transmission time slot.

FIG. 6C provides a diagrammatic representation of a retail shelving unit640 being captured by multiple systems 500 (e.g., system 500K and system500L) deployed on an opposing retail shelving unit (not shown). FIG. 6Cillustrates embodiments associated with the process of installingsystems 500 in retail store 105. To facilitate the installation ofsystem 500, each first housing 502 (e.g., first housing 502K) mayinclude an adjustment mechanism 642 for setting a field of view 644 ofimage capture device 506K such that the field of view 644 will at leastpartially encompass products placed both on a bottom shelf of retailshelving unit 640 and on a top shelf of retail shelving unit 640. Forexample, adjustment mechanism 642 may enable setting the position ofimage capture device 506K relative to first housing 502K. Adjustmentmechanism 642 may have at least two degrees of freedom to separatelyadjust manually (or automatically) the vertical field of view and thehorizontal field of view of image capture device 506K. In oneembodiment, the angle of image capture device 506K may be measured usingposition sensors associated with adjustment mechanism 642, and themeasured orientation may be used to determine if image capture device506K is positioned in the right direction. In one example, the output ofthe position sensors may be displayed on a handheld device of anemployee installing image capturing device 506K. Such an arrangement mayprovide the employee/installer with real time visual feedbackrepresentative of the field of view of an image acquisition device beinginstalled.

In addition to adjustment mechanism 642, first housing 502 may include afirst physical adapter (not shown) configured to operate with multipletypes of image capture device 506 and a second physical adapter (notshown) configured to operate with multiple types of lenses. Duringinstallation, the first physical adapter may be used to connect asuitable image capture device 506 to system 500 according to the levelof recognition requested (e.g., detecting a barcode from products,detecting text and price from labels, detecting different categories ofproducts, and so forth). Similarly, during installation, the secondphysical adapter may be used to associate a suitable lens to imagecapture device 506 according to the physical conditions at the store(e.g., the distance between the aisles, the horizontal field of viewrequired from image capture device 506, and/or the vertical field ofview required from image capture device 506). The second physicaladapter provides the employee/installer the ability to select the focallength of lens 312 during installation according to the distance betweenretail shelving units on opposite sides of an aisle (e.g., distance d1and/or distance d2 shown in FIG. 6A). In some embodiments, adjustmentmechanism 642 may include a locking mechanism to reduce the likelihoodof unintentional changes in the field of view of image capture device506. Additionally or alternatively, the at least one processor containedin second housing 504 may detect changes in the field of view of imagecapture device 506 and issue a warning when a change is detected, when achange larger than a selected threshold is detected, when a change isdetected for a duration longer than a selected threshold, and so forth.

In addition to adjustment mechanism 642 and the different physicaladapters, system 500 may modify the image data acquired by image capturedevice 506 based on at least one attribute associated with opposingretail shelving unit 640. Consistent with the present disclosure, the atleast one attribute associated with retail shelving unit 640 may includea lighting condition, the dimensions of opposing retail shelving unit640, the size of products displayed on opposing retail shelving unit640, the type of labels used on opposing retail shelving unit 640, andmore. In some embodiments, the attribute may be determined, based onanalysis of one or more acquired images, by at least one processorcontained in second housing 504. Alternatively, the attribute may beautomatically sensed and conveyed to the at least one processorcontained in second housing 504. In one example, the at least oneprocessor may change the brightness of captured images based on thedetected light conditions. In another example, the at least oneprocessor may modify the image data by cropping the image such that itwill include only the products on retail shelving unit (e.g., not toinclude the floor or the ceiling), only area of the shelving unitrelevant to a selected task (such as planogram compliance check), and soforth.

Consistent with the present disclosure, during installation, system 500may enable real-time display 646 of field of view 644 on a handhelddevice 648 of a user 650 installing image capturing device 506K. In oneembodiment, real-time display 646 of field of view 644 may includeaugmented markings 652 indicating a location of a field of view 654 ofan adjacent image capture device 506L. In another embodiment, real-timedisplay 646 of field of view 644 may include augmented markings 656indicating a region of interest in opposing retail shelving unit 640.The region of interest may be determined based on a planogram,identified product type, and/or part of retail shelving unit 640. Forexample, the region of interest may include products with a greaterlikelihood of planogram incompliance. In addition, system 500K mayanalyze acquired images to determine if field of view 644 includes thearea that image capturing device 506K is supposed to monitor (forexample, from labels on opposing retail shelving unit 640, products onopposing retail shelving unit 640, images captured from other imagecapturing devices that may capture other parts of opposing retailshelving unit 640 or capture the same part of opposing retail shelvingunit 640 but in a lower resolution or at a lower frequency, and soforth). In additional embodiments, system 500 may further comprise anindoor location sensor which may help determine if the system 500 ispositioned at the right location in retail store 105.

In some embodiments, an anti-theft device may be located in at least oneof first housing 502 and second housing 504. For example, the anti-theftdevice may include a specific RF label or a pin-tag radio-frequencyidentification device, which may be the same or similar to a type ofanti-theft device that is used by retail store 105 in which system 500is located. The RF label or the pin-tag may be incorporated within thebody of first housing 502 and second housing 504 and may not be visible.In another example, the anti-theft device may include a motion sensorwhose output may be used to trigger an alarm in the case of motion ordisturbance, in case of motion that is above a selected threshold, andso forth.

FIG. 7A includes a flowchart representing an exemplary method 700 foracquiring images of products in retail store 105 in accordance withexample embodiments of the present disclosure. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 500 as deployed in the configuration depicted inFIG. 6A. It will be appreciated, however, that other implementations arepossible and that other configurations may be utilized to implement theexemplary method. It will also be readily appreciated that theillustrated method can be altered to modify the order of steps, deletesteps, or further include additional steps.

At step 702, the method includes fixedly mounting on first retailshelving unit 602 at least one first housing 502 containing at least oneimage capture device 506 such that an optical axis (e.g., optical axis606) of at least one image capture device 506 is directed to secondretail shelving unit 604. In one embodiment, fixedly mounting firsthousing 502 on first retail shelving unit 602 may include placing firsthousing 502 on a side of store shelf 622 facing second retail shelvingunit 604. In another embodiment, fixedly mounting first housing 502 onretail shelving unit 602 may include placing first housing 502 in a slitbetween two adjacent horizontal shelves. In some embodiments, the methodmay further include fixedly mounting on first retail shelving unit 602at least one projector (such as projector 632) such that light patternsprojected by the at least one projector are directed to second retailshelving unit 604. In one embodiment, the method may include mountingthe at least one projector to first retail shelving unit 602 at aselected distance to first housing 502 with image capture device 506. Inone embodiment, the selected distance may be at least 5 cm, at least 10cm, at least 15 cm, less than 40 cm, less than 30 cm, between about 5 cmto about 20 cm, or between about 10 cm to about 15 cm. In oneembodiment, the selected distance may be calculated according to adistance between to first retail shelving unit 602 and second retailshelving unit 604, such as d1 and/or d2, for example selecting thedistance to be a function of d1 and/or d2, a linear function of d1and/or d2, a function of d1*log(d1) and/or d2*log(d2) such asa1*d1*log(d1) for some constant a1, and so forth.

At step 704, the method includes fixedly mounting on first retailshelving unit 602 second housing 504 at a location spaced apart from theat least one first housing 502, second housing 504 may include at leastone processor (e.g., processing device 302). In one embodiment, fixedlymounting second housing 504 on the retail shelving unit may includeplacing second housing 504 on a different side of store shelf 622 thanthe side first housing 502 is mounted on.

At step 706, the method includes extending at least one data conduit 508between at least one first housing 502 and second housing 504. In oneembodiment, extending at least one data conduit 508 between at least onefirst housing 502 and second housing 504 may include adjusting thelength of data conduit 508 to enable first housing 502 to be mountedseparately from second housing 504. At step 708, the method includescapturing images of second retail shelving unit 604 using at least oneimage capture device 506 contained in at least one first housing 502(e.g., first housing 502A, first housing 502B, or first housing 502C).In one embodiment, the method further includes periodically capturingimages of products located on second retail shelving unit 604. Inanother embodiment the method includes capturing images of second retailshelving unit 604 after receiving a trigger from at least one additionalsensor in communication with system 500 (wireless or wired).

At step 710, the method includes transmitting at least some of thecaptured images from second housing 504 to a remote server (e.g., server135) configured to determine planogram compliance relative to secondretail shelving unit 604. In some embodiments, determining planogramcompliance relative to second retail shelving unit 604 may includedetermining at least one characteristic of planogram compliance based ondetected differences between the at least one planogram and the actualplacement of the plurality of product types on second retail shelvingunit 604. Consistent with the present disclosure, the characteristic ofplanogram compliance may include at least one of: product facing,product placement, planogram compatibility, price correlation, promotionexecution, product homogeneity, restocking rate, and planogramcompliance of adjacent products.

FIG. 7B provides a flowchart representing an exemplary method 720 foracquiring images of products in retail store 105, in accordance withexample embodiments of the present disclosure. For purposes ofillustration, in the following description, reference is made to certaincomponents of system 500 as deployed in the configuration depicted inFIG. 6A. It will be appreciated, however, that other implementations arepossible and that other configurations may be utilized to implement theexemplary method. It will also be readily appreciated that theillustrated method can be altered to modify the order of steps, deletesteps, or further include additional steps.

At step 722, at least one processor contained in a second housing mayreceive from at least one image capture device contained in at least onefirst housing fixedly mounted on a retail shelving unit a plurality ofimages of an opposing retail shelving unit. For example, at least oneprocessor contained in second housing 504A may receive from at least oneimage capture device 506 contained in first housing 502A (fixedlymounted on first retail shelving unit 602) a plurality of images ofsecond retail shelving unit 604. The plurality of images may be capturedand collected during a period of time (e.g., a minute, an hour, sixhours, a day, a week, or more).

At step 724, the at least one processor contained in the second housingmay analyze the plurality of images acquired by the at least one imagecapture device. In one embodiment, at least one processor contained insecond housing 504A may use any suitable image analysis technique (forexample, object recognition, object detection, image segmentation,feature extraction, optical character recognition (OCR), object-basedimage analysis, shape region techniques, edge detection techniques,pixel-based detection, artificial neural networks, convolutional neuralnetworks, etc.) to identify objects in the plurality of images. In oneexample, the at least one processor contained in second housing 504A maydetermine the number of products located in second retail shelving unit604. In another example, the at least one processor contained in secondhousing 504A may detect one or more objects in an area between firstretail shelving unit 602 and second retail shelving unit 604.

At step 726, the at least one processor contained in the second housingmay identify in the plurality of images a first image that includes arepresentation of at least a portion of an object located in an areabetween the retail shelving unit and the opposing retail shelving unit.In step 728, the at least one processor contained in the second housingmay identify in the plurality of images a second image that does notinclude any object located in an area between the retail shelving unitand the opposing retail shelving unit. In one example, the object in thefirst image may be an individual, such as a customer or a storeemployee. In another example, the object in the first image may be aninanimate object, such as carts, boxes, products, etc.

At step 730, the at least one processor contained in the second housingmay instruct a network interface contained in the second housing,fixedly mounted on the retail shelving unit separate from the at leastone first housing, to transmit the second image to a remote server andto avoid transmission of the first image to the remote server. Inaddition, the at least one processor may issue a notification when anobject blocks the field of view of the image capturing device for morethan a predefined period of time (e.g., at least 30 minutes, at least 75minutes, at least 150 minutes).

Embodiments of the present disclosure may automatically assesscompliance of one or more store shelves with a planogram. For example,embodiments of the present disclosure may use signals from one or moresensors to determine placement of one or more products on store shelves.The disclosed embodiments may also use one or more sensors to determineempty spaces on the store shelves. The placements and empty spaces maybe automatically assessed against a digitally encoded planogram. Aplanogram refers to any data structure or specification that defines atleast one product characteristic relative to a display structureassociated with a retail environment (such as store shelf or area of oneor more shelves). Such product characteristics may include, among otherthings, quantities of products with respect to areas of the shelves,product configurations or product shapes with respect to areas of theshelves, product arrangements with respect to areas of the shelves,product density with respect to areas of the shelves, productcombinations with respect to areas of the shelves, etc. Althoughdescribed with reference to store shelves, embodiments of the presentdisclosure may also be applied to end caps or other displays; bins,shelves, or other organizers associated with a refrigerator or freezerunits; or any other display structure associated with a retailenvironment.

The embodiments disclosed herein may use any sensors configured todetect one or more parameters associated with products (or a lackthereof). For example, embodiments may use one or more of pressuresensors, weight sensors, light sensors, resistive sensors, capacitivesensors, inductive sensors, vacuum pressure sensors, high pressuresensors, conductive pressure sensors, infrared sensors, photo-resistorsensors, photo-transistor sensors, photo-diodes sensors, ultrasonicsensors, or the like. Some embodiments may use a plurality of differentkinds of sensors, for example, associated with the same or overlappingareas of the shelves and/or associated with different areas of theshelves. Some embodiments may use a plurality of sensors configured tobe placed adjacent a store shelf, configured for location on the storeshelf, configured to be attached to, or configured to be integrated withthe store shelf. In some cases, at least part of the plurality ofsensors may be configured to be placed next to a surface of a storeshelf configured to hold products. For example, the at least part of theplurality of sensors may be configured to be placed relative to a partof a store shelf such that the at least part of the plurality of sensorsmay be positioned between the part of a store shelf and products placedon the part of the shelf. In another embodiment, the at least part ofthe plurality of sensors may be configured to be placed above and/orwithin and/or under the part of the shelf

In one example, the plurality of sensors may include light detectorsconfigured to be located such that a product placed on the part of theshelf may block at least some of the ambient light from reaching thelight detectors. The data received from the light detectors may beanalyzed to detect a product or to identify a product based on the shapeof a product placed on the part of the shelf. In one example, the systemmay identify the product placed above the light detectors based on datareceived from the light detectors that may be indicative of at leastpart of the ambient light being blocked from reaching the lightdetectors. Further, the data received from the light detectors may beanalyzed to detect vacant spaces on the store shelf. For example, thesystem may detect vacant spaces on the store shelf based on the receiveddata that may be indicative of no product being placed on a part of theshelf. In another example, the plurality of sensors may include pressuresensors configured to be located such that a product placed on the partof the shelf may apply detectable pressure on the pressure sensors.Further, the data received from the pressure sensors may be analyzed todetect a product or to identify a product based on the shape of aproduct placed on the part of the shelf. In one example, the system mayidentify the product placed above the pressure sensors based on datareceived from the pressure sensors being indicative of pressure beingapplied on the pressure sensors. In addition, the data from the pressuresensors may be analyzed to detect vacant spaces on the store shelf, forexample based on the readings being indicative of no product beingplaced on a part of the shelf, for example, when the pressure readingsare below a selected threshold. Consistent with the present disclosure,inputs from different types of sensors (such as pressure sensors, lightdetectors, etc.) may be combined and analyzed together, for example todetect products placed on a store shelf, to identify shapes of productsplaced on a store shelf, to identify types of products placed on a storeshelf, to identify vacant spaces on a store shelf, and so forth.

With reference to FIG. 8A and consistent with the present disclosure, astore shelf 800 may include a plurality of detection elements, e.g.,detection elements 801A and 801B. In the example of FIG. 8A, detectionelements 801A and 801B may comprise pressure sensors and/or other typeof sensors for measuring one or more parameters (such as resistance,capacitance, or the like) based on physical contact (or lack thereof)with products, e.g., product 803A and product 803B. Additionally oralternatively, detection elements configured to measure one or moreparameters (such as current induction, magnetic induction, visual orother electromagnetic reflectance, visual or other electromagneticemittance, or the like) may be included to detect products based onphysical proximity (or lack thereof) to products. Consistent with thepresent disclosure, the plurality of detection elements may beconfigured for location on shelf 800. The plurality of detectionelements may be configured to detect placement of products when theproducts are placed above at least part of the plurality of detectionelements. Some embodiments of the disclosure, however, may be performedwhen at least some of the detection elements may be located next toshelf 800 (e.g., for magnetometers or the like), across from shelf 800(e.g., for image sensors or other light sensors, light detection andranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, orthe like), above shelf 800 (e.g., for acoustic sensors or the like),below shelf 800 (e.g., for pressure sensors or the like), or any otherappropriate spatial arrangement. Although depicted as standalone unitsin the example of FIG. 8A, the plurality of detection elements may formpart of a fabric (e.g., a smart fabric or the like), and the fabric maybe positioned on a shelf to take measurements. For example, two or moredetection elements may be integrated together into a single structure(e.g., disposed within a common housing, integrated together within afabric or mat, and so forth). In some examples, detection elements (suchas detection elements 801A and 801B) may be placed adjacent to (orplaced on) store shelves as described above. Some examples of detectionelements may include pressure sensors and/or light detectors configuredto be placed above and/or within and/or under a store shelf as describedabove.

Detection elements associated with shelf 800 may be associated withdifferent areas of shelf 800. For example, detection elements 801A and801B are associated with area 805A while other detection elements areassociated with area 805B. Although depicted as rows, areas 805A and805B may comprise any areas of shelf 800, whether contiguous (e.g., asquare, a rectangular, or other regular or irregular shape) or not(e.g., a plurality of rectangles or other regular and/or irregularshapes). Such areas may also include horizontal regions between shelves(as shown in FIG. 8A) or may include vertical regions that include areaof multiple different shelves (e.g., columnar regions spanning overseveral different horizontally arranged shelves). In some examples, theareas may be part of a single plane. In some examples, each area may bepart of a different plane. In some examples, a single area may be partof a single plane or be divided across multiple planes.

One or more processors (e.g., processing device 202) configured tocommunicate with the detection elements (e.g., detection elements 801Aand 801B) may detect first signals associated with a first area (e.g.,areas 805A and/or 805B) and second signals associated with a secondarea. In some embodiments, the first area may, in part, overlap with thesecond area. For example, one or more detection elements may beassociated with the first area as well as the second area and/or one ormore detection elements of a first type may be associated with the firstarea while one or more detection elements of a second type may beassociated with the second area overlapping, at least in part, the firstarea. In other embodiments, the first area and the second area may bespatially separate from each other.

The one or more processors may, using the first and second signals,determine that one or more products have been placed in the first areawhile the second area includes at least one empty area. For example, ifthe detection elements include pressure sensors, the first signals mayinclude weight signals that match profiles of particular products (suchas the mugs or plates depicted in the example of FIG. 8A), and thesecond signals may include weight signals indicative of the absence ofproducts (e.g., by being equal to or within a threshold of a defaultvalue such as atmospheric pressure or the like). The disclosed weightsignals may be representative of actual weight values associated with aparticular product type or, alternatively, may be associated with arelative weight value sufficient to identify the product and/or toidentify the presence of a product. In some cases, the weight signal maybe suitable for verifying the presence of a product regardless ofwhether the signal is also sufficient for product identification. Inanother example, if the detection elements include light detectors (asdescribed above), the first signals may include light signals that matchprofiles of particular products (such as the mugs or plates depicted inthe example of FIG. 8A), and the second signals may include lightsignals indicative of the absence of products (e.g., by being equal toor within a threshold of a default value such as values corresponding toambient light or the like). For example, the first light signals may beindicative of ambient light being blocked by particular products, whilethe second light signals may be indicative of no product blocking theambient light. The disclosed light signals may be representative ofactual light patterns associated with a particular product type or,alternatively, may be associated with light patterns sufficient toidentify the product and/or to identify the presence of a product.

The one or more processors may similarly process signals from othertypes of sensors. For example, if the detection elements includeresistive or inductive sensors, the first signals may includeresistances, voltages, and/or currents that match profiles of particularproducts (such as the mugs or plates depicted in the example of FIG. 8Aor elements associated with the products, such as tags, etc.), and thesecond signals may include resistances, voltages, and/or currentsindicative of the absence of products (e.g., by being equal to or withina threshold of a default value such as atmospheric resistance, a defaultvoltage, a default current, corresponding to ambient light, or thelike). In another example, if the detection elements include acoustics,LIDAR, RADAR, or other reflective sensors, the first signals may includepatterns of returning waves (whether sound, visible light, infraredlight, radio, or the like) that match profiles of particular products(such as the mugs or plates depicted in the example of FIG. 8A), and thesecond signals may include patterns of returning waves (whether sound,visible light, infrared light, radio, or the like) indicative of theabsence of products (e.g., by being equal to or within a threshold of apattern associated with an empty shelf or the like).

Any of the profile matching described above may include direct matchingof a subject to a threshold. For example, direct matching may includetesting one or more measured values against the profile value(s) withina margin of error; mapping a received pattern onto a profile patternwith a residual having a maximum, minimum, integral, or the like withinthe margin of error; performing an autocorrelation, Fourier transform,convolution, or other operation on received measurements or a receivedpattern and comparing the resultant values or function against theprofile within a margin of error; or the like. Additionally oralternatively, profile matching may include fuzzy matching betweenmeasured values and/or patterns and a database of profiles such that aprofile with a highest level of confidence according to the fuzzysearch. Moreover, as depicted in the example of FIG. 8A, products, suchas product 803B, may be stacked and thus associated with a differentprofile when stacked than when standalone.

Any of the profile matching described above may include use of one ormore machine learning techniques. For example, one or more artificialneural networks, random forest models, or other models trained onmeasurements annotated with product identifiers may process themeasurements from the detection elements and identify productstherefrom. In such embodiments, the one or more models may useadditional or alternative input, such as images of the shelf (e.g., fromcapturing devices 125 of FIGS. 4A-4C explained above) or the like.

Based on detected products and/or empty spaces, determined using thefirst signals and second signals, the one or more processors maydetermine one or more aspects of planogram compliance. For example, theone or more processors may identify products and their locations on theshelves, determine quantities of products within particular areas (e.g.,identifying stacked or clustered products), identify facing directionsassociated with the products (e.g., whether a product is outward facing,inward facing, askew, or the like), or the like. Identification of theproducts may include identifying a product type (e.g., a bottle of soda,a loaf of broad, a notepad, or the like) and/or a product brand (e.g., aCoca-Cola® bottle instead of a Sprite® bottle, a Starbucks® coffeetumbler instead of a Tervis® coffee tumbler, or the like). Productfacing direction and/or orientation, for example, may be determinedbased on a detected orientation of an asymmetric shape of a product baseusing pressure sensitive pads, detected density of products, etc. Forexample, the product facing may be determined based on locations ofdetected product bases relative to certain areas of a shelf (e.g., alonga front edge of a shelf), etc. Product facing may also be determinedusing image sensors, light sensors, or any other sensor suitable fordetecting product orientation.

The one or more processors may generate one or more indicators of theone or more aspects of planogram compliance. For example, an indicatormay comprise a data packet, a data file, or any other data structureindicating any variations from a planogram, e.g., with respect toproduct placement such as encoding intended coordinates of a product andactual coordinates on the shelf, with respect to product facingdirection and/or orientation such as encoding indicators of locationsthat have products not facing a correct direction and/or in an undesiredorientation, or the like.

In addition to or as an alternative to determining planogram compliance,the one or more processors may detect a change in measurements from oneor more detection elements. Such measurement changes may trigger aresponse. For example, a change of a first type may trigger capture ofat least one image of the shelf (e.g., using capturing devices 125 ofFIGS. 4A-4C explained above) while a detected change of a second typemay cause the at least one processor to forgo such capture. A first typeof change may, for example, indicate the moving of a product from onelocation on the shelf to another location such that planogram compliancemay be implicated. In such cases, it may be desired to capture an imageof the product rearrangement in order to assess or reassess productplanogram compliance. In another example, a first type of change mayindicate the removal of a product from the shelf, e.g., by an employeedue to damage, by a customer to purchase, or the like. On the otherhand, a second type of change may, for example, indicate the removal andreplacement of a product to the same (within a margin of error) locationon the shelf, e.g., by a customer to inspect the item. In cases whereproducts are removed from a shelf, but then replaced on the shelf (e.g.,within a particular time window), the system may forgo a new imagecapture, especially if the replaced product is detected in a locationsimilar to or the same as its recent, original position.

With reference to FIG. 8B and consistent with the present disclosure, astore shelf 850 may include a plurality of detection elements, e.g.,detection elements 851A and 851B. In the example of FIG. 8B, detectionelements 851A and 851B may comprise light sensors and/or other sensorsmeasuring one or more parameters (such as visual or otherelectromagnetic reflectance, visual or other electromagnetic emittance,or the like) based on electromagnetic waves from products, e.g., product853A and product 853B. Additionally or alternatively, as explained abovewith respect to FIG. 8B, detection elements 851A and 851B may comprisepressure sensors, other sensors measuring one or more parameters (suchas resistance, capacitance, or the like) based on physical contact (orlack thereof) with the products, and/or other sensors that measure oneor more parameters (such as current induction, magnetic induction,visual or other electromagnetic reflectance, visual or otherelectromagnetic emittance, or the like) based on physical proximity (orlack thereof) to products.

Moreover, although depicted as located on shelf 850, some detectionelements may be located next to shelf 850 (e.g., for magnetometers orthe like), across from shelf 850 (e.g., for image sensors or other lightsensors, light detection and ranging (LIDAR) sensors, radio detectionand ranging (RADAR) sensors, or the like), above shelf 850 (e.g., foracoustic sensors or the like), below shelf 850 (e.g., for pressuresensors, light detectors, or the like), or any other appropriate spatialarrangement. Further, although depicted as standalone in the example ofFIG. 8B, the plurality of detection elements may form part of a fabric(e.g., a smart fabric or the like), and the fabric may be positioned ona shelf to take measurements.

Detection elements associated with shelf 850 may be associated withdifferent areas of shelf 850, e.g., area 855A, area 855B, or the like.Although depicted as rows, areas 855A and 855B may comprise any areas ofshelf 850, whether contiguous (e.g., a square, a rectangular, or otherregular or irregular shape) or not (e.g., a plurality of rectangles orother regular and/or irregular shapes).

One or more processors (e.g., processing device 202) in communicationwith the detection elements (e.g., detection elements 851A and 851B) maydetect first signals associated with a first area and second signalsassociated with a second area. Any of the processing of the first andsecond signals described above with respect to FIG. 8A may similarly beperformed for the configuration of FIG. 8B.

In both FIGS. 8A and 8B, the detection elements may be integral to theshelf, part of a fabric or other surface configured for positioning onthe shelf, or the like. Power and/or data cables may form part of theshelf, the fabric, the surface, or be otherwise connected to thedetection elements. Additionally or alternatively, as depicted in FIGS.8A and 8B, individual sensors may be positioned on the shelf. Forexample, the power and/or data cables may be positioned under the shelfand connected through the shelf to the detection elements. In anotherexample, power and/or data may be transmitted wirelessly to thedetection elements (e.g., to wireless network interface controllersforming part of the detection elements). In yet another example, thedetection elements may include internal power sources (such as batteriesor fuel cells).

With reference to FIG. 9 and consistent with the present disclosure, thedetection elements described above with reference to FIGS. 8A and 8B maybe arranged on rows of the shelf in any appropriate configuration. Allof the arrangements of FIG. 9 are shown as a top-down view of a row(e.g., area 805A, area 805B, area 855A, area 855B, or the like) on theshelf. For example, arrangements 910 and 940 are both uniformdistributions of detection elements within a row. However, arrangement910 is also uniform throughout the depth of the row while arrangement940 is staggered. Both arrangements may provide signals that representproducts on the shelf in accordance with spatially uniform measurementlocations. As further shown in FIG. 9, arrangements 920, 930, 950, and960 cluster detection elements near the front (e.g., a facing portion)of the row. Arrangement 920 includes detection elements at a frontportion while arrangement 930 includes defection elements in a largerportion of the front of the shelf. Such arrangements may save power andprocessing cycles by having fewer detection elements on a back portionof the shelf Arrangements 950 and 960 include some detection elements ina back portion of the shelf but these elements are arranged less densethan detection elements in the front. Such arrangements may allow fordetections in the back of the shelf (e.g., a need to restock products, adisruption to products in the back by a customer or employee, or thelike) while still using less power and fewer processing cycles thanarrangements 910 and 940. And, such arrangements may include a higherdensity of detection elements in regions of the shelf (e.g., a frontedge of the shelf) where product turnover rates may be higher than inother regions (e.g., at areas deeper into a shelf), and/or in regions ofthe shelf where planogram compliance is especially important.

FIG. 10A is a flow chart, illustrating an exemplary method 1000 formonitoring planogram compliance on a store shelf, in accordance with thepresently disclosed subject matter. It is contemplated that method 1000may be used with any of the detection element arrays discussed abovewith reference to, for example, FIGS. 8A, 8B and 9. The order andarrangement of steps in method 1000 is provided for purposes ofillustration. As will be appreciated from this disclosure, modificationsmay be made to process 1000, for example, adding, combining, removing,and/or rearranging one or more steps of process 1000.

Method 1000 may include a step 1005 of receiving first signals from afirst subset of detection elements (e.g., detection elements 801A and801B of FIG. 8A) from among the plurality of detection elements afterone or more of a plurality of products (e.g., products 803A and 803B)are placed on at least one area of the store shelf associated with thefirst subset of detection elements. As explained above with respect toFIGS. 8A and 8B, the plurality of detection elements may be embeddedinto a fabric configured to be positioned on the store shelfAdditionally or alternatively, the plurality of detection elements maybe configured to be integrated with the store shelf. For example, anarray of pressure sensitive elements (or any other type of detector) maybe fabricated as part of the store shelf. In some examples, theplurality of detection elements may be configured to placed adjacent to(or located on) store shelves, as described above.

As described above with respect to arrangements 910 and 940 of FIG. 9,the plurality of detection elements may be substantially uniformlydistributed across the store shelf Alternatively, as described abovewith respect to arrangements 920, 930, 950, and 960 of FIG. 9, theplurality of detection elements may be distributed relative to the storeshelf such that a first area of the store shelf has a higher density ofdetection elements than a second area of the store shelf. For example,the first area may comprise a front portion of the shelf, and the secondarea may comprise a back portion of the shelf.

In some embodiments, such as those including pressure sensors or othercontact sensors as depicted in the example of FIG. 8A, step 1005 mayinclude receiving the first signals from the first subset of detectionelements as the plurality of products are placed above the first subsetof detection elements. In some embodiments where the plurality ofdetection elements includes pressure detectors, the first signals may beindicative of pressure levels detected by pressure detectorscorresponding to the first subset of detection elements after one ormore of the plurality of products are placed on the at least one area ofthe store shelf associated with the first subset of detection elements.For example, the first signals may be indicative of pressure levelsdetected by pressure detectors corresponding to the first subset ofdetection elements after stocking at least one additional product abovea product previously positioned on the shelf, removal of a product fromthe shelf, or the like. In other embodiments where the plurality ofdetection elements includes light detectors, the first signals may beindicative of light measurements made with respect to one or more of theplurality of products placed on the at least one area of the store shelfassociated with the first subset of detection elements. Specifically,the first signals may be indicative of at least part of the ambientlight being blocked from reaching the light detectors by the one or moreof the plurality of products.

In embodiments including proximity sensors as depicted in the example ofFIG. 8B, step 1005 may include receiving the first signals from thefirst subset of detection elements as the plurality of products areplaced below the first subset of detection elements. In embodimentswhere the plurality of detection elements include proximity detectors,the first signals may be indicative of proximity measurements made withrespect to one or more of the plurality of products placed on the atleast one area of the store shelf associated with the first subset ofdetection elements.

Method 1000 may include step 1010 of using the first signals to identifyat least one pattern associated with a product type of the plurality ofproducts. For example, any of the pattern matching techniques describedabove with respect to FIGS. 8A and 8B may be used for identification. Apattern associated with a product type may include a pattern (e.g., acontinuous ring, a discontinuous ring of a certain number of points, acertain shape, etc.) associated with a base of a single product. Thepattern associated with a product type may also be formed by a group ofproducts. For example, a six pack of soda cans may be associated with apattern including a 2×3 array of continuous rings associated with thesix cans of that product type. Additionally, a grouping of two literbottles may form a detectable pattern including an array (whetheruniform, irregular, or random) of discontinuous rings of pressurepoints, where the rings have a diameter associated with a particular2-liter product. Various other types of patterns may also be detected(e.g., patterns associated with different product types arrangedadjacent to one another, patterns associated with solid shapes (such asa rectangle of a boxed product), and so forth). In another example, anartificial neural network configured to recognize product types may beused to analyze the signals received by step 1005 (such as signals frompressure sensors, from light detectors, from contact sensors, and soforth) to determine product types associated with products placed on anarea of a shelf (such as an area of a shelf associated with the firstsubset of detection elements). In yet another example, a machinelearning algorithm trained using training examples to recognize producttypes may be used to analyze the signals received by step 1005 (such assignals from pressure sensors, from light detectors, from contactsensors, and so forth) to determine product types associated withproducts placed on an area of a shelf (such as an area of a shelfassociated with the first subset of detection elements).

In some embodiments, step 1010 may further include accessing a memorystoring data (e.g., memory device 226 of FIG. 2 and/or memory device 314of FIG. 3A) associated with patterns of different types of products. Insuch embodiments, step 1010 may include using the first signals toidentify at least one product of a first type using a first pattern (ora first product model) and at least one product of a second type using asecond pattern (or a second product model). For example, the first typemay include one brand (such as Coca-Cola® or Folgers®) while the secondtype may include another brand (such as Pepsi® or Maxwell House®). Inthis example, a size, shape, point spacing, weight, resistance or otherproperty of the first brand may be different from that of the secondbrand such that the detection elements may differentiate the brands.Such characteristics may also be used to differentiate like-branded, butdifferent products from one another (e.g., a 12-ounce can of Coca Cola,versus a 16 oz bottle of Coca Cola, versus a 2-liter bottle of CocaCola). For example, a soda may have a base detectable by a pressuresensitive pad as a continuous ring. Further, the can of soda may beassociated with a first weight signal having a value recognizable asassociated with such a product. A 16 ounce bottle of soda may beassociated with a base having four or five pressure points, which apressure sensitive pad may detect as arranged in a pattern associatedwith a diameter typical of such a product. The 16 ounce bottle of sodamay also be associated with a second weight signal having a value higherthan the weight signal associated with the 12 ounce can of soda. Furtherstill, a 2 liter bottle of soda may be associated with a base having aring, four or five pressure points, etc. that a pressure sensitive padmay detect as arranged in a pattern associated with a diameter typicalof such a product. The 2 liter bottle of soda may be associated with aweight signal having a value higher than the weight signal associatedwith the 12 ounce can of soda and 16 ounce bottle of soda.

In the example of FIG. 8B, the different bottoms of product 853A andproduct 853B may be used to differentiate the products from each other.For example, detection elements such as pressure sensitive pads may beused to detect a product base shape and size (e.g., ring, pattern ofpoints, asymmetric shape, base dimensions, and so forth). Such a baseshape and size may be used (optionally, together with one or more weightsignals) to identify a particular product. The signals may also be usedto identify and/or distinguish product types from one another. Forexample, a first type may include one category of product (such as sodacans) while a second type may include a different category of product(such as notepads). In another example, detection elements such as lightdetectors may be used to detect a product based on a pattern of lightreadings indicative of a product blocking at least part of the ambientlight from reaching the light detectors. Such pattern of light readingsmay be used to identify product type and/or product category and/orproduct shape. For example, products of a first type may block a firstsubset of light frequencies of the ambient light from reaching the lightdetectors, while products of a second type may block a second subset oflight frequencies of the ambient light from reaching the light detectors(the first subset and second subset may differ). In this case the typeof the products may be determined based on the light frequenciesreaching the light detectors. In another example, products of a firsttype may have a first shape of shades and therefore may block ambientlight from reaching light detectors arranged in one shape, whileproducts of a second type may have a second shape of shades andtherefore may block ambient light from reaching light detectors arrangedin another shape. In this case the type of the products may bedetermined based on the shape of blocked ambient light. Any of thepattern matching techniques described above may be used for theidentification.

Additionally or alternatively, step 1010 may include using the at leastone pattern to determine a number of products placed on the at least onearea of the store shelf associated with the first subset of detectionelements. For example, any of the pattern matching techniques describedabove may be used to identify the presence of one or more product typesand then to determine the number of products of each product type (e.g.,by detecting a number of similarly sized and shaped product bases andoptionally by detecting weight signals associated with each detectedbase). In another example, an artificial neural network configured todetermine the number of products of selected product types may be usedto analyze the signals received by step 1005 (such as signals frompressure sensors, from light detectors, from contact sensors, and soforth) to determine the number of products of selected product typesplaced on an area of a shelf (such as an area of a shelf associated withthe first subset of detection elements). In yet another example, amachine learning algorithm trained using training examples to determinethe number of products of selected product types may be used to analyzethe signals received by step 1005 (such as signals from pressuresensors, from light detectors, from contact sensors, and so forth) todetermine the number of products of selected product types placed on anarea of a shelf (such as an area of a shelf associated with the firstsubset of detection elements). Additionally or alternatively, step 1010may include extrapolating from a stored pattern associated with a singleproduct (or type of product) to determine the number of productsmatching the first signals. In such embodiments, step 1010 may furtherinclude determining, for example based on product dimension data storedin a memory, a number of additional products that can be placed on theat least one area of the store shelf associated with the second subsetof detection elements. For example, step 1010 may include extrapolatingbased on stored dimensions of each product and stored dimensions of theshelf area to determine an area and/or volume available for additionalproducts. Step 1010 may further include extrapolation of the number ofadditional products based on the stored dimensions of each product anddetermined available area and/or volume.

Method 1000 may include step 1015 of receiving second signals from asecond subset of detection elements (e.g., detection elements 851A and851B of FIG. 8B) from among the plurality of detection elements, thesecond signals being indicative of no products being placed on at leastone area of the store shelf associated with the second subset ofdetection elements. Using this information, method 1000 may include step1020 of using the second signals to determine at least one empty spaceon the store shelf. For example, any of the pattern matching techniquesdescribed above may be used to determine that the second signals includedefault values or other values indicative of a lack of product incertain areas associated with a retail store shelf. A default value maybe include, for example, a pressure signal associated with an un-loadedpressure sensor or pressure sensitive mat, indicating that no product islocated in a certain region of a shelf In another example, a defaultvalue may include signals from light detectors corresponding to ambientlight, indicating that no product is located in a certain region of ashelf

Method 1000 may include step 1025 of determining, based on the at leastone pattern associated with a detected product and the at least oneempty space, at least one aspect of planogram compliance. As explainedabove with respect to FIGS. 8A and 8B, the aspect of planogramcompliance may include the presence or absence of particular products(or brands), locations of products on the shelves, quantities ofproducts within particular areas (e.g., identifying stacked or clusteredproducts), facing directions associated with the products (e.g., whethera product is outward facing, inward facing, askew, or the like), or thelike. A planogram compliance determination may be made, for example, bydetermining a number of empty spaces on a shelf and determining alocation of the empty spaces on a shelf. The planogram determination mayalso include determining weight signal magnitudes associated withdetected products at the various detected non-empty locations. Thisinformation may be used by the one or more processors in determiningwhether a product facing specification has been satisfied (e.g., whethera front edge of a shelf has a suitable number of products or suitabledensity of products), whether a specified stacking density has beenachieved (e.g., by determining a pattern of detected products and weightsignals of the detected products to determine how many products arestacked at each location), whether a product density specification hasbeen achieved (e.g., by determining a ratio of empty locations toproduct-present locations), whether products of a selected product typeare located in a selected area of the shelf, whether all productslocated in a selected area of the shelf are of a selected product type,whether a selected number of products (or a selected number of productsof a selected product type) are located in a selected area of the shelf,whether products located in a selected area of a shelf are positioned ina selected orientation, or whether any other aspect of one or moreplanograms has been achieved.

For example, the at least one aspect may include product homogeneity,and step 1025 may further include counting occurrences where a productof the second type is placed on an area of the store shelf associatedwith the first type of product. For example, by accessing a memoryincluding base patterns (or any other type of pattern associated withproduct types, such as product models), the at least one processor maydetect different products and product types. A product of a first typemay be recognized based on a first pattern, and product of a second typemay be recognized based on a second, different pattern (optionally alsobased on weight signal information to aid in differentiating betweenproducts). Such information may be used, for example, to monitor whethera certain region of a shelf includes an appropriate or intended productor product type. Such information may also be useful in determiningwhether products or product types have been mixed (e.g., producthomogeneity). Regarding planogram compliance, detection of differentproducts and their relative locations on a shelf may aid in determiningwhether a product homogeneity value, ratio, etc. has been achieved. Forexample, the at least one processor may count occurrences where aproduct of a second type is placed on an area of the store shelfassociated with a product of a first type.

Additionally or alternatively, the at least one aspect of planogramcompliance may include a restocking rate, and step 1025 may furtherinclude determining the restocking rate based on a sensed rate at whichproducts are added to the at least one area of the store shelfassociated with the second subset of detection elements. Restocking ratemay be determined, for example, by monitoring a rate at which detectionelement signals change as products are added to a shelf (e.g., whenareas of a pressure sensitive pad change from a default value to aproduct-present value).

Additionally or alternatively, the at least one aspect of planogramcompliance may include product facing, and step 1025 may further includedetermining the product facing based on a number of products determinedto be placed on a selected area of the store shelf at a front of thestore shelf. Such product facing may be determined by determining anumber of products along a certain length of a front edge of a storeshelf and determining whether the number of products complies with, forexample, a specified density of products, a specified number ofproducts, and so forth.

Step 1025 may further include transmitting an indicator of the at leastone aspect of planogram compliance to a remote server. For example, asexplained above with respect to FIGS. 8A and 8B, the indicator maycomprise a data packet, a data file, or any other data structureindicating any variations from a planogram, e.g., with respect toproduct (or brand) placement, product facing direction, or the like. Theremote server may include one or more computers associated with a retailstore (e.g., so planogram compliance may be determined on a local basiswithin a particular store), one or more computers associated with aretail store evaluation body (e.g., so planogram compliance may bedetermined across a plurality of retail stores), one or more computersassociated with a product manufacturer, one or more computers associatedwith a supplier (such as supplier 115), one or more computers associatedwith a market research entity (such as market research entity 110), etc.

Method 1000 may further include additional steps. For example, method1000 may include identifying a change in at least one characteristicassociated with one or more of the first signals (e.g., signals from afirst group or type of detection elements), and in response to theidentified change, triggering an acquisition of at least one image ofthe store shelf. The acquisition may be implemented by activating one ormore of capturing devices 125 of FIGS. 4A-4C, as explained above. Forexample, the change in at least one characteristic associated with oneor more of the first signals may be indicative of removal of at leastone product from a location associated with the at least one area of thestore shelf associated with the first subset of detection elements.Accordingly, method 1000 may include triggering the acquisition todetermine whether restocking, reorganizing, or other intervention isrequired, e.g., to improve planogram compliance. Thus, method 1000 mayinclude identifying a change in at least one characteristic associatedwith one or more of the first signals; and in response to the identifiedchange, trigger a product-related task for an employee of the retailstore.

Additionally or alternatively, method 1000 may be combined with method1050 of FIG. 10B, described below, such that step 1055 is performed anytime after step 1005.

FIG. 10B is a flow chart, illustrating an exemplary method 1050 fortriggering image capture of a store shelf, in accordance with thepresently disclosed subject matter. It is contemplated that method 1050may be used in conjunction with any of the detection element arraysdiscussed above with reference to, for example, FIGS. 8A, 8B and 9. Theorder and arrangement of steps in method 1050 is provided for purposesof illustration. As will be appreciated from this disclosure,modifications may be made to process 1050, for example, adding,combining, removing, and/or rearranging one or more steps of process1050.

Method 1050 may include a step 1055 of determining a change in at leastone characteristic associated with one or more first signals. Forexample, the first signals may have been captured as part of method 1000of FIG. 10A, described above. For example, the first signals may includepressure readings when the plurality of detection elements includespressure sensors, contact information when the plurality of detectionelements includes contact sensors, light readings when the plurality ofdetection elements includes light detectors (for example, from lightdetectors configured to be placed adjacent to (or located on) a surfaceof a store shelf configured to hold products, as described above), andso forth.

Method 1050 may include step 1060 of using the first signals to identifyat least one pattern associated with a product type of the plurality ofproducts. For example, any of the pattern matching techniques describedabove with respect to FIGS. 8A, 8B, and step 1010 may be used foridentification.

Method 1050 may include step 1065 of determining a type of eventassociated with the change. For example, a type of event may include aproduct removal, a product placement, movement of a product, or thelike.

Method 1050 may include step 1070 of triggering an acquisition of atleast one image of the store shelf when the change is associated with afirst event type. For example, a first event type may include removal ofa product, moving of a product, or the like, such that the first eventtype may trigger a product-related task for an employee of the retailstore depending on analysis of the at least one image. The acquisitionmay be implemented by activating one or more of capturing devices 125 ofFIGS. 4A-4C, as explained above. In some examples, the triggeredacquisition may include an activation of at least one projector (such asprojector 632). In some examples, the triggered acquisition may includeacquisition of color images, depth images, stereo images, active stereoimages, time of flight images, LIDAR images, RADAR images, and so forth.

Method 1050 may include a step (not shown) of forgoing the acquisitionof at least one image of the store shelf when the change is associatedwith a second event type. For example, a second event type may includereplacement of a removed product by a customer, stocking of a shelf byan employee, or the like. As another example, a second event type mayinclude removal, placement, or movement of a product that is detectedwithin a margin of error of the detection elements and/or detectedwithin a threshold (e.g., removal of only one or two products; movementof a product by less than 5 cm, 20 cm, or the like; moving of a facingdirection by less than 10 degrees; or the like), such that no imageacquisition is required.

FIGS. 11A-11E illustrate example outputs based on data automaticallyderived from machine processing and analysis of images captured inretail store 105 according to disclosed embodiments. FIG. 11Aillustrates an optional output for market research entity 110. FIG. 11Billustrates an optional output for supplier 115. FIGS. 11C and 11Dillustrate optional outputs for employees of retail store 105. And FIG.11E illustrates optional outputs for user 120.

FIG. 11A illustrates an example graphical user interface (GUI) 500 foroutput device 145A, representative of a GUI that may be used by marketresearch entity 110. Consistent with the present disclosure, marketresearch entity 110 may assist supplier 115 and other stakeholders inidentifying emerging trends, launching new products, and/or developingmerchandising and distribution plans across a large number of retailstores 105. By doing so, market research entity 110 may assist supplier115 in growing product presence and maximizing or increasing new productsales. As mentioned above, market research entity 110 may be separatedfrom or part of supplier 115. To successfully launch a new product,supplier 115 may use information about what really happens in retailstore 105. For example, supplier 115 may want to monitor how marketingplans are being executed and to learn what other competitors are doingrelative to certain products or product types. Embodiments of thepresent disclosure may allow market research entity 110 and suppliers115 to continuously monitor product-related activities at retail stores105 (e.g., using system 100 to generate various metrics or informationbased on automated analysis of actual, timely images acquired from theretail stores). For example, in some embodiments, market research entity110 may track how quickly or at what rate new products are introduced toretail store shelves, identify new products introduced by variousentities, assess a supplier's brand presence across different retailstores 105, among many other potential metrics.

In some embodiments, server 135 may provide market research entity 110with information including shelf organization, analysis of skewproductivity trends, and various reports aggregating information onproducts appearing across large numbers of retail stores 105. Forexample, as shown in FIG. 11A, GUI 1100 may include a first display area1102 for showing a percentage of promotion campaign compliance indifferent retail stores 105. GUI 1100 may also include a second displayarea 1104 showing a graph illustrating sales of a certain productrelative to the percentage of out of shelf. GUI 1100 may also include athird display area 1106 showing actual measurements of different factorsrelative to target goals (e.g., planogram compliance, restocking rate,price compliance, and other metrics). The provided information mayenable market research entity 110 to give supplier 115 informed shelvingrecommendations and fine-tune promotional strategies according toin-store marketing trends, to provide store managers with a comparisonof store performances in comparison to a group of retail stores 105 orindustry wide performances, and so forth.

FIG. 11B illustrates an example GUI 1110 for output device 145B used bysupplier 115. Consistent with the present disclosure, server 135 may usedata derived from images captured in a plurality of retail stores 105 torecommend a planogram, which often determines sales success of differentproducts. Using various analytics and planogram productivity measures,server 135 may help supplier 115 to determine an effective planogramwith assurances that most if not all retail stores 105 can execute theplan. For example, the determined planogram may increase the probabilitythat inventory is available for each retail store 105 and may bedesigned to decrease costs or to keep costs within a budget (such asinventory costs, restocking costs, shelf space costs, and so forth).Server 135 may also provide pricing recommendations based on the goalsof supplier 115 and other factors. In other words, server 135 may helpsupplier 115 understand how much room to reserve for different productsand how to make them available for favorable sales and profit impact(for example, by choosing the size of the shelf dedicated to a selectedproduct, the location of the shelf, the height of the shelf, theneighboring products, and so forth). In addition, server 135 may monitornear real-time data from retail stores 105 to determine or confirm thatretail stores 105 are compliant with the determined planogram ofsupplier 115. As used herein, the term “near real-time data,” in thecontext of this disclosure, refers to data acquired or generated, etc.,based on sensor readings and other inputs (such as data from imagesensors, audio sensors, pressure sensors, checkout stations, etc.) fromretail store 105 received by system 100 within a predefined period oftime (such as time periods having durations of less than a second, lessthan a minute, less than an hour, less than a day, less than a week, andso forth).

In some embodiments, server 135 may generate reports that summarizeperformance of the current assortment and the planogram compliance.These reports may advise supplier 115 of the category and the itemperformance based on individual SKU, sub segments of the category,vendor, and region. In addition, server 135 may provide suggestions orinformation upon which decisions may be made regarding how or when toremove markdowns and when to replace underperforming products. Forexample, as shown in FIG. 11B, GUI 1110 may include a first display area1112 for showing different scores of supplier 115 relative to scoresassociated with its competitors. GUI 1110 may also include a seconddisplay area 1114 showing the market share of each competitor. GUI 1110may also include a third display area 1116 showing retail measurementsand distribution of brands. GUI 1110 may also include a fourth displayarea 1118 showing a suggested planogram. The provided information mayhelp supplier 115 to select preferred planograms based on projected orobserved profitability, etc., and to ensure that retail stores 105 arefollowing the determined planogram.

FIGS. 11C and 11D illustrate example GUIs for output devices 145C, whichmay be used by employees of retail store 105. FIG. 11C depicts a GUI1120 for a manager of retail store 105 designed for a desktop computer,and FIG. 11D depicts GUI 1130 and 1140 for store staff designed for ahandheld device. In-store execution is one of the challenges retailstores 105 have in creating a positive customer experience. Typicalin-store execution may involve dealing with ongoing service events, suchas a cleaning event, a restocking event, a rearrangement event, andmore. In some embodiments, system 100 may improve in-store execution byproviding adequate visibility to ensure that the right products arelocated at preferred locations on the shelf For example, using nearreal-time data (e.g., captured images of store shelves) server 135 maygenerate customized online reports. Store managers and regionalmanagers, as well as other stakeholders, may access custom dashboardsand online reports to see how in-store conditions (such as, planogramcompliance, promotion compliance, price compliance, etc.) are affectingsales. This way, system 100 may enable managers of retail stores 105 tostay on top of burning issues across the floor and assign employees toaddress issues that may negatively impact the customer experience.

In some embodiments, server 135 may cause real-time automated alertswhen products are out of shelf (or near out of shelf), when pricing isinaccurate, when intended promotions are absent, and/or when there areissues with planogram compliance, among others. In the example shown inFIG. 11C, GUI 1120 may include a first display area 1122 for showing theaverage scores (for certain metrics) of a specific retail store 105 overa selected period of time. GUI 1120 may also include a second displayarea 1124 for showing a map of the specific retail store 105 withreal-time indications of selected in-store execution events that requireattention, and a third display area 1126 for showing a list of theselected in-store execution events that require attention. In anotherexample, shown in FIG. 11D, GUI 1130 may include a first display area1132 for showing a list of notifications or text messages indicatingselected in-store execution events that require attention. Thenotifications or text messages may include a link to an image (or theimage itself) of the specific aisle with the in-store execution event.In another example, shown in FIG. 11D, GUI 1140 may include a firstdisplay area 1142 for showing a display of a video stream captured byoutput device 145C (e.g., a real-time display or a near real-timedisplay) with augmented markings indicting a status of planogramcompliance for each product (e.g., correct place, misplaced, not inplanogram, empty, and so forth). GUI 1140 may also include a seconddisplay area 1144 for showing a summary of the planogram compliance forall the products identified in the video stream captured by outputdevice 145C. Consistent with the present disclosure, server 135 maygenerate within minutes actionable tasks to improve store execution.These tasks may help employees of retail store 105 to quickly addresssituations that can negatively impact revenue and customer experience inthe retail store 105.

FIG. 11E illustrates an example GUI 1150 for output device 145D used byan online customer of retail store 105. Traditional online shoppingsystems present online customers with a list of products. Productsselected for purchase may be placed into a virtual shopping cart untilthe customers complete their virtual shopping trip. Virtual shoppingcarts may be examined at any time, and their contents can be edited ordeleted. However, common problems of traditional online shopping systemsarise when the list of products on the website does not correspond withthe actual products on the shelf. For example, an online customer mayorder a favorite cookie brand without knowing that the cookie brand isout-of-stock. Consistent with some embodiments, system 100 may use imagedata acquired by capturing devices 125 to provide the online customerwith a near real-time display of the retail store and a list of theactual products on the shelf based on near real-time data. In oneembodiment, server 135 may select images without occlusions in the fieldof view (e.g., without other customers, carts, etc.) for the nearreal-time display. In one embodiment, server 135 may blur or erasedepictions of customers and other people from the near real-timedisplay. As used herein, the term “near real-time display,” in thecontext of this disclosure, refers to image data captured in retailstore 105 that was obtained by system 100 within a predefined period oftime (such as less than a second, less than a minute, less than about 30minutes, less than an hour, less than 3 hours, or less than 12 hours)from the time the image data was captured.

Consistent with the present disclosure, the near real-time display ofretail store 105 may be presented to the online customer in a mannerenabling easy virtual navigation in retail store 105. For example, asshown in FIG. 11E, GUI 1150 may include a first display area 1152 forshowing the near real-time display and a second display area 1154 forshowing a product list including products identified in the nearreal-time display. In some embodiments, first display area 1152 mayinclude different GUI features (e.g., tabs 1156) associated withdifferent locations or departments of retail store 105. By selectingeach of the GUI features, the online customer can virtually jump todifferent locations or departments in retail store 105. For example,upon selecting the “bakery” tab, GUI 1150 may present a near real-timedisplay of the bakery of retail store 105. In addition, first displayarea 1152 may include one or more navigational features (e.g., arrows1158A and 1158B) for enabling the online customer to virtually movewithin a selected department and/or virtually walk through retail store105. Server 135 may be configured to update the near real-time displayand the product list upon determining that the online customer wants tovirtually move within retail store 105. For example, after identifying aselection of arrow 1158B, server 135 may present a different section ofthe dairy department and may update the product list accordingly. Inanother example, server 135 may update the near-real time display andthe product list in response to new captured images and new informationreceived from retail store 105. Using GUI 1150, the online customer mayhave the closest shopping experience without actually being in retailstore 105. For example, an online customer can visit the vegetabledepartment and decide not to buy tomatoes after seeing that they are notripe enough.

In some retail stores, selecting which information to present, as wellas where and how to present it, may increase productivity, among otherpotential benefits. Consist with the present disclosure, such selectionmay be based on actual current and past inventory and condition ofproducts in selected parts of a retail store (such as aisle, shelf,retail storage container, and so forth).

FIG. 12 is a block diagram representative of an example configuration ofelectronic visual display control system 1200. In one embodiment,electronic visual display control system 1200 may include a bus 200 (orany other communication mechanism) that interconnects subsystems andcomponents for transferring information within electronic visual displaycontrol system 1200. For example, bus 200 may interconnect a processingdevice 202, a memory interface 204, a network interface 206, and aperipherals interface 208 connected to an I/O system 210.

In one implementation of electronic visual display control system 1200,I/O system 210 may include an electronic visual display controller 1212,an audio controller 214, and/or other input controller(s) 216.Electronic visual display controller 1212 may be coupled to one or moreelectronic visual displays (such as touch screen 218, electronic visualdisplay 1306, electronic visual display 1322, electronic visual display1324, electronic visual display 1342, and so forth). In one example,electronic visual display controller 1212 may include touch screencontroller 212.

In one implementation of electronic visual display control system 1200,processing device 202 may use memory interface 204 to access data and asoftware product stored on a memory device 1226. Memory device 1226 mayinclude operating system programs for electronic visual display controlsystem 1200 that perform operating system functions when executed by theprocessing device.

Memory device 1226 may also store communication instructions 228,graphical user interface instructions 230, image processing instructions232, sensor processing instructions 234, web browsing instructions 236,and other software instructions 238 to facilitate other processes andfunctions. Memory device 1226 may also store product type model data240, catalog data 244, inventory data 246, employee data 248, andcalendar data 250.

In one embodiment, memory device 1226 may also store display rules 1242that may be used to determine which information to present, as well aswhere and how to present it, for example based on actual current andpast inventory and condition of products in selected parts of a retailstore (such as aisle, shelf, retail storage container, and so forth).

The components and arrangements shown in FIG. 12 are not intended tolimit the disclosed embodiments. As will be appreciated by a personskilled in the art having the benefit of this disclosure, numerousvariations and/or modifications may be made to the depictedconfiguration of electronic visual display control system 1200 and tothe content of memory device 1226. For example, components may beremoved, modified and/or added to electronic visual display controlsystem 1200 and/or to memory device 1200. In another example, componentsof electronic visual display control system 1200 may be distributedacross different systems. In yet another example, each component ofelectronic visual display control system 1200, including memory device1226 may be distributed across different systems. For example, not allcomponents may be essential for the operation of electronic visualdisplay control system 1200 in all cases. Any component may be locatedin any appropriate part of electronic visual display control system1200, and the components may be rearranged into a variety ofconfigurations while providing the functionality of the disclosedembodiments.

FIG. 13A is a schematic cross-sectional side view illustration of anexemplary door 1300 for a retail storage container, consistent with thepresent disclosure. In this example, door 1300 may comprise an outersurface 1304, a connector 1302 to an electronic visual displaycontroller, and an electronic visual display 1306. The components andarrangements shown in FIG. 13A are not intended to limit the disclosedembodiments. As will be appreciated by a person skilled in the arthaving the benefit of this disclosure, numerous variations and/ormodifications may be made to the depicted configuration of door 1300.For example, connector 1302 may further include or be replaced by anelectronic visual display control system (such as electronic visualdisplay control system 1200). In another example, door 1300 may furtherinclude a power source and/or a connector to an external power source.

FIG. 13B is a schematic cross-sectional side view illustration of anexemplary door 1320 for a retail storage container, consistent with thepresent disclosure. In this example, door 1320 may comprise an outersurface 1304, a connector 1302 to an electronic visual displaycontroller, an electronic visual display 1322, an electronic visualdisplay 1324, and thermal insulation 1326. The components andarrangements shown in FIG. 13B are not intended to limit the disclosedembodiments. As will be appreciated by a person skilled in the arthaving the benefit of this disclosure, numerous variations and/ormodifications may be made to the depicted configuration of door 1320.For example, connector 1302 may further include or be replaced by anelectronic visual display control system (such as electronic visualdisplay control system 1200). In another example, door 1320 may furtherinclude a power source and/or a connector to an external power source.In yet another example, at least one of electronic visual display 1322,electronic visual display 1324 and insulation 1326 may be removed fromdoor 1320.

In some examples, side 1310 of doors 1300 and 1320 may be configured toface the internal side of the retail storage container when the door isclosed. In some examples, side 1312 of doors 1300 and 1320 may beconfigured to face customers when the door is closed (i.e., to faceoutwards from the retail storage container when the door is closed).

FIG. 13C is a schematic cross-sectional view illustration of anexemplary door 1340 for a retail storage container, consistent with thepresent disclosure. In this example, door 1340 may comprise an outersurface 1304, a connector 1302 to an electronic visual displaycontroller, and an electronic visual display 1342. The components andarrangements shown in FIG. 13B are not intended to limit the disclosedembodiments. As will be appreciated by a person skilled in the arthaving the benefit of this disclosure, numerous variations and/ormodifications may be made to the depicted configuration of door 1340.For example, connector 1302 may further include or be replaced by anelectronic visual display control system (such as electronic visualdisplay control system 1200). In another example, door 1340 may furtherinclude a power source and/or a connector to an external power source.

In one example, any one of doors 1300, 1320 and 1340 may be a slidingdoor, may be a hinged door, and so forth. In one example, parts of outersurface 1304 may be opaque, may be transparent, may be partlytransparent, may be covered by a mirror, may comprise an electronicvisual display, and so forth. For example, in door 1300, outer surface1304 may include transparent or partly transparent portions that enablea person to see electronic visual display 1306 through outer surface1304.

In another example, in door 1300, outer surface 1304 may include opaqueportions that hide at least part of connector 1302 from a person lookingat the door. In one example, parts of outer surface 1304 may include oneor more holes or niches. For example, in door 1320, outer surface 1304may include a hole or a niche for electronic visual display 1322, mayinclude a hole or a niche for electronic visual display 1324, and soforth. In one example, connector 1302 may be configured to connect to anelectronic visual display control system (such as electronic visualdisplay control system 1200). In one example, connector 1302 may furtherinclude or be replaced by at least parts of an electronic visual displaycontrol system (such as electronic visual display control system 1200).

In one example, an electronic visual display (such as electronic visualdisplays 1306, 1322, 1324 and 1342) may include any electronic devicefor visually displaying visual information, such as text, images,videos, and so forth. Some non-limiting examples of such electronicdevices may include touch screens, flat panel displays, non-flat paneldisplays, electroluminescent displays, liquid-crystal displays (LCD),light-emitting diode (LED) displays, active-matrix organiclight-emitting diode (AMOLED) displays, organic light-emitting diode(OLED) displays, plasma displays, quantum displays, micro-LED displays,and so forth. In some examples, an electronic visual display consistentwith the present disclosure may be part of or connected to at least oneof a door of a retail storage container, a retail shelf, a fixed windowof a retail storage container, a fixed insulated glass end-window of aretail storage container, a fixed window of a walk-in retail storagecontainer, a mobile device, a personal device, and so forth.

In one example, causing an electronic visual display (such as electronicvisual displays 1306, 1322, 1324 and 1342) to display information (forexample by steps 1810, 1812, 1910 and 2010) may include providing data(for example, by transmitting the data, by storing the data in a sharedmemory, etc.) that is configured to cause the electronic visual displayto display the information. In another example, causing an electronicvisual display (such as electronic visual displays 1306, 1322, 1324 and1342) to display information may include using electronic visual displaycontrol system and/or electronic visual display controller 1212, forexample by providing instructions, to cause the electronic visualdisplay to display the information. In some examples, causing anelectronic visual display (such as electronic visual displays 1306,1322, 1324 and 1342) to display information (for example by step 2010)may include causing the electronic visual display to display informationaccording to a selected at least one display parameter. For example,data (for example, by transmitting the data, by storing the data in ashared memory, etc.) that is configured to cause the electronic visualdisplay to display information using the at least one display parametermay be provided to the electronic visual display, to a systemcontrolling the electronic visual display (such as electronic visualdisplay control system 1200, electronic visual display controller 1212,and so forth). In another example, a visual (such as image, video, 2Dvisual, 3D visual, etc.) may be generated based on the using the atleast one display parameter, and the electronic visual display may becaused to display the generated visual as described above.

In some examples, causing an adjustment to a power scheme of anelectronic visual display (such as electronic visual displays 1306,1322, 1324 and 1342) may comprise changing the brightness of theelectronic visual display, turning the electronic visual display on,turning the electronic visual display off, and so forth. In one example,causing an adjustment to a power scheme of an electronic visual display(such as electronic visual displays 1306, 1322, 1324 and 1342) maycomprise providing data that is configured to cause the adjustment tothe power scheme of the electronic visual display. In another example,causing an adjustment to a power scheme of an electronic visual display(such as electronic visual displays 1306, 1322, 1324 and 1342) maycomprise using electronic visual display control system and/orelectronic visual display controller 1212, for example by providinginstructions, to cause the electronic visual display to adjust the powerscheme of the electronic visual display.

Each one of FIG. 14A-14F illustrates an example of a retail storagecontainer with an open hinged door, and each one of FIG. 15A-15Hillustrates an example of a retail storage container with a closedhinged door. The illustrated retail storage containers may compriseshelves that hold products. While FIG. 14A-14F and FIG. 15A-15H depict aspecific type of retail storage containers for purposes of illustration,as will be appreciated from this disclosure, other types of retailstorage containers that include doors may be used. Some non-limitingexamples of such retail storage container may include a cooler (such asa reach-in cooler, walk-in cooler, display cooler, countertop cooler,under-counter cooler, worktop cooler, chest cooler, merchandisingcooler, etc.), a refrigerator (such as a reach-in refrigerator, displayrefrigerator, walk-in refrigerator, countertop refrigerator,under-counter refrigerator, worktop refrigerator, chest refrigerator,merchandising refrigerator, etc.), a freezer (such as a reach-infreezer, walk-in freezer, display freezer, countertop freezer,under-counter freezer, worktop freezer, chest freezer, merchandisingfreezer, etc.), a closet, enclosed storage unit with a door, shelvingunit with a door, or any other unit configured to include at least onedoor and is configured to hold one or more products for sale in a retailestablishment. Some examples of doors of retail storage containers mayinclude a sliding door, a hinged door, and so forth. In some examples,the door may be an integral door of the retail storage container. Insome examples, such door of a retail storage container may comprise atleast an external part that is configured to face customers when thedoor is closed and an internal part configured to face the internal sideof the retail storage container when the door is closed (for example ina hinged door).

FIG. 14A-14F are schematic illustrations of exemplary retail storagecontainers, consistent with the present disclosure.

In FIG. 14A, at least a portion of the internal part of the door may beopaque, may be transparent, may be partly transparent, may be covered bya mirror, may comprise an electronic visual display, and so forth.

In FIG. 14B, the internal part of the door may comprise an electronicvisual display, and the electronic visual display may be configured todisplay promotional information (such as ‘50% off’, ‘special price’,‘limited offer’, ‘buy one get one free’, image of a product beingpromoted, name of product being promoted, and so forth). In someexamples, the displayed promotional information may be selected and/orcontrolled as described herein.

In FIG. 14C, the internal part of the door may comprise an electronicvisual display, and the electronic visual display may display one ormore instructions for store associates (such as ‘remove X items ofproduct Y’, ‘restock product Y’, ‘reposition product Y’, and so forth),for example as described herein. In some examples, the displayed one ormore instructions for the store associates may be selected and/orcontrolled as described herein. In some examples, the electronic visualdisplay may include a touch screen, and clicking on an instruction maycause a change in the displayed information, for example as describedherein.

In FIG. 14D and in FIG. 14E, the internal part of the door may comprisean electronic visual display, and the electronic visual display maydisplay information about products associated with the retail storagecontainer (such as products in the retail storage container, productmissing from the retail storage container, and so forth), for example asdescribed herein.

In FIG. 14F, the internal part of the door may comprise a touch screen,and the touch screen may display a user interface that enables a user(such as a customer, a shop associate, and so forth) to interact withthe system. In this example, the touch screen may display an image of aproduct missing from the retail storage container, for example with atext ‘Click to Order’, and clicking on the image and/or clicking on thetext may trigger an action associate with ordering the missing product.

In FIG. 15A, at least a portion of the door may be transparent and/orpartly transparent, and shelves and/or products in the retail storagecontainer may be visible and/or partly visible to a person facing theretail storage container through the door (for example, through a closeddoor, through a partly closed door, and so forth).

In FIG. 15B, the external part of the door may comprise an electronicvisual display. In this example, the electronic visual display maydisplay information about products associated with the retail storagecontainer (such as products in the retail storage container, productmissing from the retail storage container, and so forth), for example asdescribed herein. Further, in this example, the electronic visualdisplay may display promotional information (such as ‘50% off’, ‘specialprice’, ‘limited offer’, ‘buy one get one free’, image of a productbeing promoted, name of product being promoted, and so forth), forexample as described herein. In some examples, the displayed promotionalinformation may be selected and/or controlled as described herein.

In FIG. 15C, at least a portion of the door may comprise a transparentelectronic visual display and/or partly transparent electronic visualdisplay, and shelves and/or products in the retail storage container maybe visible and/or partly visible to a person facing the retail storagecontainer through the electronic visual display (for example, through aclosed door, through a partly closed door, and so forth). Further, inthis example, the transparent electronic visual display and/or thepartly transparent electronic visual display may display informationabout products associated with the retail storage container (such asproducts in the retail storage container, product missing from theretail storage container, and so forth), for example as describedherein. For example, an overlay displayed over the products and/orshelves in the retail storage container may present information relatedto the overlaid products and/or shelves, for example as describedherein. In another example, an overlay displayed over empty spaces inthe retail storage container may present information related to missingproducts, for example as described herein.

In yet another example, an overlay displayed over empty spaces in theretail storage container may present promotional information (such as‘50% off’, ‘special price’, ‘limited offer’, ‘buy one get one free’,image of a product being promoted, name of product being promoted, andso forth), for example as described herein. In some examples, thedisplayed promotional information may be selected and/or controlled asdescribed herein.

In FIG. 15D, at least a portion of the door may comprise a transparentelectronic visual display and/or partly transparent electronic visualdisplay, and shelves and/or products in the retail storage container maybe visible and/or partly visible to a person facing the retail storagecontainer through the electronic visual display (for example, through aclosed door, through a partly closed door, and so forth). Further, inthis example, the transparent electronic visual display and/or thepartly transparent electronic visual display may display promotionalinformation (such as ‘50% off’, ‘special price’, ‘limited offer’, ‘buyone get one free’, image of a product being promoted, name of productbeing promoted, and so forth), for example as described herein. In someexamples, the displayed promotional information may be selected and/orcontrolled as described herein.

In FIG. 15E, 15F and 15G, the external part of the door may comprise anelectronic visual display. In this example, the electronic visualdisplay may display information about products associated with theretail storage container (such as products in the retail storagecontainer, product missing from the retail storage container, and soforth), for example as described herein.

In FIG. 15H, at least a portion of the door may comprise an electronicvisual display and/or a transparent electronic visual display and/orpartly transparent electronic visual display, and in someimplementations shelves and/or products in the retail storage containermay be visible and/or partly visible to a person facing the retailstorage container through the electronic visual display (for example,through a closed door, through a partly closed door, and so forth).Further, in this example, the electronic visual display and/or thetransparent electronic visual display and/or the partly transparentelectronic visual display may display one or more instructions for storeassociates (such as ‘remove X items of product Y’, ‘restock product Y’,‘reposition product Y’, and so forth), for example as described herein.In some examples, the displayed one or more instructions for the storeassociates may be selected and/or controlled as described herein. Insome examples, the electronic visual display may include a touch screen,and clicking on an instruction may cause a change in the displayedinformation, for example as described herein.

FIG. 16A-16F are schematic illustrations of exemplary retail shelves,consistent with the present disclosure. Each one of FIG. 16A-16Fillustrates an example of a retail shelf 1602 that holds one or moreproducts in a retail store, and an associated electronic visual display1604. While FIG. 16A-16F depict a specific type of retail shelf 1602 forpurposes of illustration, as will be appreciated from this disclosure,other types of units for holding products in a retail store may be used.Some non-limiting examples of such units may include any type of shelve,any type of shelving unit, a display, any type of retail storagecontainer, and so forth. Moreover, while FIG. 16A-16F depict electronicvisual display 1604 physically connected to retail shelf 1602 forpurposes of illustration, as will be appreciated from this disclosure,electronic visual display 1604 may be physically disconnected fromretail shelf 1602. For example, electronic visual display 1604 may beconnected to another retail shelf or another retail unit, may be placedon a stand, may be part of a mobile device, and so forth.

In FIG. 16A, electronic visual display 1604 may display promotionalinformation (such as ‘50% off’, ‘special price’, ‘limited offer’, ‘buyone get one free’, image of a product being promoted, name of productbeing promoted, and so forth), for example as described herein. In someexamples, the displayed promotional information may be selected and/orcontrolled as described herein.

In FIG. 16B, 16C and 16D, electronic visual display 1604 may displayinformation about products associated with shelf 1602 (such as productson shelf 1602, product missing from shelf 1602, and so forth), forexample as described herein.

In FIG. 16E, electronic visual display 1604 may display one or moreinstructions for store associates (such as ‘remove X items of productY’, ‘restock product Y’, ‘reposition product Y’, and so forth) , forexample as described herein. In some examples, the displayed one or moreinstructions for the store associates may be selected and/or controlledas described herein. In some examples, electronic visual display 1604may include a touch screen, and clicking on an instruction may cause achange in the displayed information, for example as described herein.

In FIG. 16F, electronic visual display 1604 may comprise a touch screen,and the touch screen may display a user interface that enables a user(such as a customer, a shop associate, and so forth) to interact withthe system. In this example, the touch screen may display an image of aproduct missing from the retail storage container, for example with atext ‘Click to Order’, and clicking on the image and/or clicking on thetext may trigger an action associate with ordering the missing product.

In some embodiments, a method (such as methods 700, 720, 1000, 1050,1700, 1800, 1900, 2000, 2100, 2200, etc.) may comprise of one or moresteps. In some examples, a method, as well as all individual stepstherein, may be performed by various aspects of server 135, capturingdevice 125, electronic visual display control system 1200, and so forth.For example, the method may be performed by processing units (such asprocessors 202) executing software instructions stored within memoryunits (such as memory device 226, memory device 1226, and so forth). Insome examples, a method, as well as all individual steps therein, may beperformed by a dedicated hardware. In some examples, computer readablemedium (such as a non-transitory computer readable medium) may storedata and/or computer implementable instructions for carrying out amethod, such as instructions that when executed by a processor may causethe processor to perform the method. Some non-limiting examples ofpossible execution manners of a method may include continuous execution(for example, returning to the beginning of the method once the methodnormal execution ends), periodically execution, executing the method atselected times, execution upon the detection of a trigger (somenon-limiting examples of such trigger may include a trigger from a user,a trigger from another method, a trigger from an external device, etc.),and so forth.

In some embodiments, machine learning algorithms (also referred to asmachine learning models in the present disclosure) may be trained usingtraining examples, for example in the cases described below. Somenon-limiting examples of such machine learning algorithms may includeclassification algorithms, data regressions algorithms, imagesegmentation algorithms, visual detection algorithms (such as objectdetectors, face detectors, person detectors, motion detectors, edgedetectors, etc.), visual recognition algorithms (such as facerecognition, person recognition, object recognition, etc.), speechrecognition algorithms, mathematical embedding algorithms, naturallanguage processing algorithms, support vector machines, random forests,nearest neighbors algorithms, deep learning algorithms, artificialneural network algorithms, convolutional neural network algorithms,recurrent neural network algorithms, linear algorithms, non-linearalgorithms, ensemble algorithms, and so forth. For example, a trainedmachine learning algorithm may comprise an inference model, such as apredictive model, a classification model, a regression model, aclustering model, a segmentation model, an artificial neural network(such as a deep neural network, a convolutional neural network, arecurrent neural network, etc.), a random forest, a support vectormachine, and so forth. In some examples, the training examples mayinclude example inputs together with the desired outputs correspondingto the example inputs. Further, in some examples, training machinelearning algorithms using the training examples may generate a trainedmachine learning algorithm, and the trained machine learning algorithmmay be used to estimate outputs for inputs not included in the trainingexamples. In some examples, engineers, scientists, processes andmachines that train machine learning algorithms may further usevalidation examples and/or test examples. For example, validationexamples and/or test examples may include example inputs together withthe desired outputs corresponding to the example inputs, a trainedmachine learning algorithm and/or an intermediately trained machinelearning algorithm may be used to estimate outputs for the exampleinputs of the validation examples and/or test examples, the estimatedoutputs may be compared to the corresponding desired outputs, and thetrained machine learning algorithm and/or the intermediately trainedmachine learning algorithm may be evaluated based on a result of thecomparison. In some examples, a machine learning algorithm may haveparameters and hyper parameters, where the hyper parameters are setmanually by a person or automatically by an process external to themachine learning algorithm (such as a hyper parameter search algorithm),and the parameters of the machine learning algorithm are set by themachine learning algorithm according to the training examples. In someimplementations, the hyper-parameters are set according to the trainingexamples and the validation examples, and the parameters are setaccording to the training examples and the selected hyper-parameters.

In some embodiments, trained machine learning algorithms (also referredto as trained machine learning models in the present disclosure) may beused to analyze inputs and generate outputs, for example in the casesdescribed below. In some examples, a trained machine learning algorithmmay be used as an inference model that when provided with an inputgenerates an inferred output. For example, a trained machine learningalgorithm may include a classification algorithm, the input may includea sample, and the inferred output may include a classification of thesample (such as an inferred label, an inferred tag, and so forth). Inanother example, a trained machine learning algorithm may include aregression model, the input may include a sample, and the inferredoutput may include an inferred value for the sample. In yet anotherexample, a trained machine learning algorithm may include a clusteringmodel, the input may include a sample, and the inferred output mayinclude an assignment of the sample to at least one cluster. In anadditional example, a trained machine learning algorithm may include aclassification algorithm, the input may include an image, and theinferred output may include a classification of an item depicted in theimage. In yet another example, a trained machine learning algorithm mayinclude a regression model, the input may include an image, and theinferred output may include an inferred value for an item depicted inthe image (such as an estimated property of the item, such as size,volume, age of a person depicted in the image, cost of a productdepicted in the image, and so forth). In an additional example, atrained machine learning algorithm may include an image segmentationmodel, the input may include an image, and the inferred output mayinclude a segmentation of the image. In yet another example, a trainedmachine learning algorithm may include an object detector, the input mayinclude an image, and the inferred output may include one or moredetected objects in the image and/or one or more locations of objectswithin the image. In some examples, the trained machine learningalgorithm may include one or more formulas and/or one or more functionsand/or one or more rules and/or one or more procedures, the input may beused as input to the formulas and/or functions and/or rules and/orprocedures, and the inferred output may be based on the outputs of theformulas and/or functions and/or rules and/or procedures (for example,selecting one of the outputs of the formulas and/or functions and/orrules and/or procedures, using a statistical measure of the outputs ofthe formulas and/or functions and/or rules and/or procedures, and soforth).

In some embodiments, artificial neural networks may be configured toanalyze inputs and generate corresponding outputs. Some non-limitingexamples of such artificial neural networks may comprise shallowartificial neural networks, deep artificial neural networks, feedbackartificial neural networks, feed forward artificial neural networks,autoencoder artificial neural networks, probabilistic artificial neuralnetworks, time delay artificial neural networks, convolutionalartificial neural networks, recurrent artificial neural networks, longshort term memory artificial neural networks, and so forth. In someexamples, an artificial neural network may be configured manually. Forexample, a structure of the artificial neural network may be selectedmanually, a type of an artificial neuron of the artificial neuralnetwork may be selected manually, a parameter of the artificial neuralnetwork (such as a parameter of an artificial neuron of the artificialneural network) may be selected manually, and so forth. In someexamples, an artificial neural network may be configured using a machinelearning algorithm. For example, a user may select hyper-parameters forthe an artificial neural network and/or the machine learning algorithm,and the machine learning algorithm may use the hyper-parameters andtraining examples to determine the parameters of the artificial neuralnetwork, for example using back propagation, using gradient descent,using stochastic gradient descent, using mini-batch gradient descent,and so forth. In some examples, an artificial neural network may becreated from two or more other artificial neural networks by combiningthe two or more other artificial neural networks into a singleartificial neural network.

In some embodiments, analyzing one or more images (for example, by themethods, steps and modules described herein) may comprise analyzing theone or more images to obtain a preprocessed image data, and subsequentlyanalyzing the one or more images and/or the preprocessed image data toobtain the desired outcome. One of ordinary skill in the art willrecognize that the followings are examples, and that the one or moreimages may be preprocessed using other kinds of preprocessing methods.In some examples, the one or more images may be preprocessed bytransforming the one or more images using a transformation function toobtain a transformed image data, and the preprocessed image data maycomprise the transformed image data. For example, the transformed imagedata may comprise one or more convolutions of the one or more images.For example, the transformation function may comprise one or more imagefilters, such as low-pass filters, high-pass filters, band-pass filters,all-pass filters, and so forth. In some examples, the transformationfunction may comprise a nonlinear function. In some examples, the one ormore images may be preprocessed by smoothing at least parts of the oneor more images, for example using Gaussian convolution, using a medianfilter, and so forth. In some examples, the one or more images may bepreprocessed to obtain a different representation of the one or moreimages. For example, the preprocessed image data may comprise: arepresentation of at least part of the one or more images in a frequencydomain; a Discrete Fourier Transform of at least part of the one or moreimages; a Discrete Wavelet Transform of at least part of the one or moreimages; a time/frequency representation of at least part of the one ormore images; a representation of at least part of the one or more imagesin a lower dimension; a lossy representation of at least part of the oneor more images; a lossless representation of at least part of the one ormore images; a time ordered series of any of the above; any combinationof the above; and so forth. In some examples, the one or more images maybe preprocessed to extract edges, and the preprocessed image data maycomprise information based on and/or related to the extracted edges. Insome examples, the one or more images may be preprocessed to extractimage features from the one or more images. Some non-limiting examplesof such image features may comprise information based on and/or relatedto: edges; corners; blobs; ridges; Scale Invariant Feature Transform(SIFT) features; temporal features; and so forth.

In some embodiments, analyzing one or more images (for example, by themethods, steps and modules described herein) may comprise analyzing theone or more images and/or the preprocessed image data using one or morerules, functions, procedures, artificial neural networks, objectdetection algorithms, face detection algorithms, visual event detectionalgorithms, action detection algorithms, motion detection algorithms,background subtraction algorithms, inference models, and so forth. Somenon-limiting examples of such inference models may include: an inferencemodel preprogrammed manually; a classification model; a regressionmodel; a result of training algorithms, such as machine learningalgorithms and/or deep learning algorithms, on training examples, wherethe training examples may include examples of data instances, and insome cases, a data instance may be labeled with a corresponding desiredlabel and/or result; and so forth.

In some embodiments, analyzing one or more images (for example, by themethods, steps and modules described herein) may comprise analyzingpixels, voxels, point cloud, range data, etc. included in the one ormore images.

FIG. 17 provides a flowchart of an exemplary method 1700 for controllinginformation displayed on an electronic visual display in a retail store,consistent with the present disclosure. In this example, method 1700 maycomprise: receiving information from one or more sensors (step 1702);analyzing the information received from the one or more sensors todetermine information related to products in a retail store (step 1704);analyzing the information received from the one or more sensors todetermine information related to one or more people in a vicinity of anelectronic visual display (step 1706); and using the determinedinformation related to products in a retail store and/or the determinedinformation related to one or more people to control informationdisplayed on the electronic visual display (step 1708). In one example,step 1704 may be omitted from method 1700, and step 1708 may use thedetermined information related to one or more people to controlinformation displayed on the electronic visual display. In anotherexample, step 1706 may be omitted from method 1700, and step 1708 mayuse the determined information related to products in a retail store tocontrol information displayed on the electronic visual display. Somenon-limiting examples of such electronic visual display may includetouch screen, electronic visual display 1306, electronic visual display1322, electronic visual display 1324, electronic visual display 1342,any one of the electronic visual display in FIG. 14A-14F, any one of theelectronic visual display in FIG. 15A-15H, any one of the electronicvisual display in FIG. 16A-16F, and so forth.

In some embodiments, step 1702 may comprise receiving information fromone or more sensors. For example, step 1702 may use one or more of steps708, 722, 1005, 1015, 1802, 1804, 1902 and 2102 to obtain theinformation from one or more sensors. In one example, step 1702 mayobtain one or more images captured using one or more capturing devices125. In another example, step 1702 may obtain one or more imagescaptured as described in relation to FIG. 4A and/or in relation to FIG.4B and/or in relation to FIG. 4C and/or in relation to FIG. 5A and/or inrelation to FIG. 5B and/or in relation to FIG. 5C and/or in relation toFIG. 6A and/or in relation to FIG. 6B and/or in relation to FIG. 6C maybe obtained. In yet another example, step 1702 may obtain one or morereadings from sensors configured to be positioned between a retail shelfand products placed on the retail shelf, for example as described inrelation to FIG. 8A and/or in relation to FIG. 8B and/or in relation toFIG. 9. Some non-limiting examples of such sensors may include pressuresensors, touch sensors, weight sensors, light sensors, resistivesensors, capacitive sensors, inductive sensors, vacuum pressure sensors,high pressure sensors, conductive pressure sensors, infrared sensors,photo-resistor sensors, photo-transistor sensors, photo-diodes sensors,ultrasonic sensors, and so forth. For example, step 1702 may comprisereceiving pressure data captured using pressure sensors configured to bepositioned between a retail shelf and products placed on the retailshelf. In another example, step 1702 may comprise receiving touch datacaptured using touch sensors configured to be positioned between aretail shelf and products placed on the retail shelf. In yet anotherexample, step 1702 may comprise receiving weight data captured usingweight sensors configured to be positioned between a retail shelf andproducts placed on the retail shelf. In an additional example, step 1702may comprise receiving light data captured using light sensorsconfigured to be positioned between a retail shelf and products placedon the retail shelf

In some embodiments, step 1704 may comprise analyzing the informationreceived from the one or more sensors by step 1702 to determineinformation related to products in a retail store (for example, todetermine information related to products in retail storage container,to determine information related to products placed on a retail shelf,and so forth). In some examples, step 1704 may use the analysis of theinformation received by step 1702 to determine the types of theproducts, the placement of the products, the amount of the products, thecondition and/or state of the products, and so forth. For example, step1704 may use one or more of steps 724, 1010, 1020, 1025, 1055, 1060,1904, 1906 and 2104 to analyze the information received by step 1702 anddetermine the information related to products in the retail store. Inanother example, a machine learning algorithm may be trained usingtraining examples to determine information about products from suchinformation, and step 1704 may use the trained machine learning model toanalyze the information received by step 1702 and determine theinformation related to products in the retail store. An example of suchtraining example may include a sample of a received input data togetherwith desired determined information related to products. In anotherexample, an artificial neural network (such as a deep neural network, aconvolutional neural network, etc.) may be configured to determineinformation related to products from such received information, and step1704 may use the artificial neural network to analyze the informationreceived by step 1702 and determine the information related to productsin the retail store.

In some embodiments, step 1706 may comprise analyzing the informationreceived from the one or more sensors by step 1702 to determineinformation related to one or more people in a vicinity of an electronicvisual display. For example, step 1706 may obtain a location of a personthrough a localization of a personalized device associated with theperson (such as a smartphone, wearable device, etc.) within the retailstore, through person and/or face detection in images captured from theenvironment surrounding the electronic visual display, and so forth. Inanother example, step 1706 may obtain the identity and/or other personalinformation of a person in the vicinity of the electronic visual displayfrom the personalized device associated with the person, through facerecognition, from a loyalty plan of a customer, from past purchases ofthe customer, from an employee record of a store associate, and soforth. In yet another example, step 1706 may obtain information about astate and/or actions of the person (such as emotional state, interactionwith at least part of the electronic visual display, picking of aproduct, returning of a product, etc.) by analyzing images captured fromthe environment surrounding the electronic visual display. For example,step 1706 may use face recognition algorithms to recognize a person inan image captured from the environment of the electronic visual display,and use the recognition of the person to access a record correspondingto the person that contains at least part of the information related tothe person. In another example, step 1706 may use age and/or genderestimation algorithms to estimate an age and/or a gender of a person inan image captured from the environment of the electronic visual display.In yet another example, step 1706 may receive from a personal device ofa person a wireless communication including a unique identifier (such asa MAC address, a loyalty card number, an employee number, etc.)corresponding to the personal device and/or to the person, and step 1706may use the unique identifier to access a database including a recordwith at least part of the information related to the person. In anadditional information, step 1706 may use tracking algorithms todetermine past behavior of the person. In yet another example, step 1706may use image analysis algorithm to determine sentiment and/or emotionalstate of the person.

In some embodiments, step 1708 may comprise using the informationrelated to products in a retail store determined by step 1704 and/or theinformation related to one or more people determined by step 1706 tocontrol information displayed on the electronic visual display. Forexample, step 1708 may use one or more of methods 1800, 1900, 2000, 2100and 2200 to control information displayed on the electronic visualdisplay.

In some embodiments, step 1708 may select and/or modify promotionalinformation displayed on an electronic visual display (such as thedisplayed promotional information in FIG. 14B, FIG. 15C, FIG. 15D andFIG. 16A) in response to external triggers, in response to actualinventory (in a retail storage container, on a retail shelf, etc.), inresponse to a planogram (of a retail storage container, of a retailshelf, etc.), in response to a realogram (of a retail storage container,of a retail shelf, etc.), in response to a state of at least one product(in a retail storage container, on a retail shelf, etc.), in response tosupply chain information, in response to an action (such as looking at aproduct, clicking at a touch screen and/or a key, picking a product,returning a product, etc.) of a person (such as a customer, a storeassociate, etc.), in response to information (such as identity of theperson, age of the person, gender of the person, past behavior of theperson, sentiment and/or emotional state of the person, etc.) on aperson (such as a customer, a store associate, etc.), and so forth. Somenon-limiting examples of such promotional information may include anindication of a discount (for example, a percentage discount, a flatamount discount, etc.), an indication of a multi-buy promotion (such asa buy-one-get-one promotion, a “two for the price of one” promotion,etc.), an indication of a multi-save promotion, an indication of aconditional promotion, a free-shipping promotion, a try-before-you-buypromotion, and so forth. For example, in response to a first externaltrigger, step 1708 may cause first promotional information to bedisplayed on the electronic visual display, and in response to a secondexternal trigger, step 1708 may cause second promotional information tobe displayed on the electronic visual display. In another example, inresponse to a first external trigger, step 1708 may cause firstpromotional information to be displayed on the electronic visualdisplay, and in response to a second external trigger, step 1708 mayforgo and/or withhold causing the display of the first promotionalinformation. For example, in response to a first actual inventory (inthe retail storage container, on the shelf, etc.), step 1708 may causefirst promotional information to be displayed on the electronic visualdisplay, and in response to a second actual inventory, step 1708 maycause second promotional information to be displayed on the electronicvisual display. In another example, in response to a first actualinventory (in the retail storage container, on the shelf, etc.), step1708 may cause first promotional information to be displayed on theelectronic visual display, and in response to a second actual inventory,step 1708 may forgo and/or withhold causing the display of the firstpromotional information. For example, in response to a first planogram(of the retail storage container, of the shelf, etc.), step 1708 maycause first promotional information to be displayed on the electronicvisual display, and in response to a second planogram, step 1708 maycause second promotional information to be displayed on the electronicvisual display. In another example, in response to a first planogram (ofthe retail storage container, of the shelf, etc.), step 1708 may causefirst promotional information to be displayed on the electronic visualdisplay, and in response to a second planogram, step 1708 may forgoand/or withhold causing the display of the first promotionalinformation. For example, in response to a first realogram (of theretail storage container, of the shelf, etc.), step 1708 may cause firstpromotional information to be displayed on the electronic visualdisplay, and in response to a second realogram, step 1708 may causesecond promotional information to be displayed on the electronic visualdisplay. In another example, in response to a first realogram (of theretail storage container, of the shelf, etc.), step 1708 may cause firstpromotional information to be displayed on the electronic visualdisplay, and in response to a second realogram, step 1708 may forgoand/or withhold causing the display of the first promotionalinformation. For example, in response to a first state of the at leastone product (in the retail storage container, on the shelf, etc.), step1708 may cause first promotional information to be displayed on theelectronic visual display, and in response to a second state of the atleast one product, step 1708 may cause second promotional information tobe displayed on the electronic visual display. In another example, inresponse to a first state of the at least one product (in the retailstorage container, on the shelf, etc.), step 1708 may cause firstpromotional information to be displayed on the electronic visualdisplay, and in response to a second state of the at least one product,step 1708 may forgo and/or withhold causing the display of the firstpromotional information. For example, in response to first supply chaininformation, step 1708 may cause first promotional information to bedisplayed on the electronic visual display, and in response to secondsupply chain information, step 1708 may cause second promotionalinformation to be displayed on the electronic visual display. In anotherexample, in response to first supply chain information, step 1708 maycause first promotional information to be displayed on the electronicvisual display, and in response to second supply chain information, step1708 may forgo and/or withhold causing the display of the firstpromotional information. For example, in response to a first action(such as looking at a product, clicking at a touch screen and/or a key,picking a product, returning a product, etc.) of a person (such as acustomer, a store associate, etc.), step 1708 may cause firstpromotional information to be displayed on the electronic visualdisplay, and in response to a second action of the person, step 1708 maycause second promotional information to be displayed on the electronicvisual display. In another example, in response to a first action (suchas looking at a product, clicking at a touch screen and/or a key,picking a product, returning a product, etc.) of a person (such as acustomer, a store associate, etc.), step 1708 may cause firstpromotional information to be displayed on the electronic visualdisplay, and in response to a second action of the person, step 1708 mayforgo and/or withhold causing the display of the first promotionalinformation. For example, in response to first information (such asidentity, age, gender, past behavior, sentiment and/or emotional state,etc.) on a person (such as a customer, a store associate, etc.), step1708 may cause first promotional information to be displayed on theelectronic visual display, and in response to second information on theperson, step 1708 may cause second promotional information to bedisplayed on the electronic visual display. In another example, inresponse to first information (such as identity, age, gender, pastbehavior, sentiment and/or emotional state, etc.) of a person (such as acustomer, a store associate, etc.), step 1708 may cause firstpromotional information to be displayed on the electronic visualdisplay, and in response to second information on the person, step 1708may forgo and/or withhold causing the display of the first promotionalinformation. The second promotional information may differ from thefirst promotional information. In some examples, the electronic visualdisplay may be a touch screen, and clicking on the promotionalinformation may cause the electronic visual display to displayadditional information, may cause transmission of information to anexternal system, and so forth.

In some embodiments, step 1708 may select and/or modify one or moreinstructions to one or more store associates displayed on an electronicvisual display (such as displayed instructions in FIG. 14C, FIG. 15H andFIG. 16E) in response to external triggers, in response to actualinventory (in a retail storage container, on a retail shelf, etc.), inresponse to a planogram (of a retail storage container, of a retailshelf, etc.), in response to a realogram (of a retail storage container,of a retail shelf, etc.), in response to a state of at least one product(in a retail storage container, on a retail shelf, etc.), in response tosupply chain information, in response to an action (such as looking at aproduct, clicking at a touch screen and/or a key, picking a product,returning a product, etc.) of a person (such as a customer, a storeassociate, etc.), in response to information (such as identity of theperson, age of the person, gender of the person, past behavior of theperson, sentiment and/or emotional state of the person, etc.) on aperson (such as a customer, a store associate, etc.), and so forth. Somenon-limiting examples of such instructions for the store associates mayinclude an instruction to restock products, an instruction to rearrangeproducts, an instruction to remove products, an instruction to clean, aninstruction to modify a label, an instruction to place a label, aninstruction to remove a label, and so forth. For example, in response toa first external trigger, step 1708 may cause a first instruction forthe store associates to be displayed on the electronic visual display,and in response to a second external trigger, step 1708 may cause asecond instruction for the store associates to be displayed on theelectronic visual display. In another example, in response to a firstexternal trigger, step 1708 may cause a first instruction for the storeassociates to be displayed on the electronic visual display, and inresponse to a second external trigger, step 1708 may forgo and/orwithhold causing the display of the first instruction for the storeassociates. For example, in response to a first actual inventory (in theretail storage container, on the shelf, etc.), step 1708 may cause afirst instruction for the store associates to be displayed on theelectronic visual display, and in response to a second actual inventory,step 1708 may cause a second instruction for the store associates to bedisplayed on the electronic visual display. In another example, inresponse to a first actual inventory (in the retail storage container,on the shelf, etc.), step 1708 may cause a first instruction for thestore associates to be displayed on the electronic visual display, andin response to a second actual inventory, step 1708 may forgo and/orwithhold causing the display of the first instruction for the storeassociates. For example, in response to a first planogram (of the retailstorage container, of the shelf, etc.), step 1708 may cause a firstinstruction for the store associates to be displayed on the electronicvisual display, and in response to a second planogram, step 1708 maycause a second instruction for the store associates to be displayed onthe electronic visual display. In another example, in response to afirst planogram (of the retail storage container, of the shelf, etc.),step 1708 may cause a first instruction for the store associates to bedisplayed on the electronic visual display, and in response to a secondplanogram, step 1708 may forgo and/or withhold causing the display ofthe first instruction for the store associates. For example, in responseto a first realogram (of the retail storage container, of the shelf,etc.), step 1708 may cause a first instruction for the store associatesto be displayed on the electronic visual display, and in response to asecond realogram, step 1708 may cause a second instruction for the storeassociates to be displayed on the electronic visual display. In anotherexample, in response to a first realogram (of the retail storagecontainer, of the shelf, etc.), step 1708 may cause a first instructionfor the store associates to be displayed on the electronic visualdisplay, and in response to a second realogram, step 1708 may forgoand/or withhold causing the display of the first instruction for thestore associates. For example, in response to a first state of the atleast one product (in the retail storage container, on the shelf, etc.),step 1708 may cause a first instruction for the store associates to bedisplayed on the electronic visual display, and in response to a secondstate of the at least one product, step 1708 may cause a secondinstruction for the store associates to be displayed on the electronicvisual display. In another example, in response to a first state of theat least one product (in the retail storage container, on the shelf,etc.), step 1708 may cause a first instruction for the store associatesto be displayed on the electronic visual display, and in response to asecond state of the at least one product, step 1708 may forgo and/orwithhold causing the display of the first instruction for the storeassociates. For example, in response to first supply chain information,step 1708 may cause a first instruction for the store associates to bedisplayed on the electronic visual display, and in response to secondsupply chain information, step 1708 may cause a second instruction forthe store associates to be displayed on the electronic visual display.In another example, in response to first supply chain information, step1708 may cause a first instruction for the store associates to bedisplayed on the electronic visual display, and in response to secondsupply chain information, step 1708 may forgo and/or withhold causingthe display of the first instruction for the store associates. Forexample, in response to a first action (such as looking at a product,clicking at a touch screen and/or a key, picking a product, returning aproduct, etc.) of a person (such as a customer, a store associate,etc.), step 1708 may cause a first instruction for the store associatesto be displayed on the electronic visual display, and in response to asecond action of the person, step 1708 may cause a second instructionfor the store associates to be displayed on the electronic visualdisplay. In another example, in response to a first action (such aslooking at a product, clicking at a touch screen and/or a key, picking aproduct, returning a product, etc.) of a person (such as a customer, astore associate, etc.), step 1708 may cause a first instruction for thestore associates to be displayed on the electronic visual display, andin response to second action of the person, step 1708 may forgo and/orwithhold causing the display of the first instruction for the storeassociates. For example, in response to first information (such asidentity, age, gender, past behavior, sentiment and/or emotional state,etc.) on a person (such as a customer, a store associate, etc.), step1708 may cause a first instruction for the store associates to bedisplayed on the electronic visual display, and in response to secondinformation on the person, step 1708 may cause a second instruction forthe store associates to be displayed on the electronic visual display.In another example, in response to first information (such as identity,age, gender, past behavior, sentiment and/or emotional state, etc.) of aperson (such as a customer, a store associate, etc.), step 1708 maycause a first instruction for the store associates to be displayed onthe electronic visual display, and in response to second information onthe person, step 1708 may forgo and/or withhold causing the display ofthe first instruction for the store associates. The second instructionfor the store associates may differ from the first instruction for thestore associates. In some examples, the electronic visual display may bea touch screen, and clicking on an instruction may cause the electronicvisual display to display additional information, may transmitinformation to an external system, may remove the instruction from thedisplayed information, and so forth.

In some embodiments, step 1708 may select and/or modify elements of auser interface displayed on an electronic visual display (such aselements of the user interface in FIG. 14F and FIG. 16F) in response toexternal triggers, in response to actual inventory (in a retail storagecontainer, on a retail shelf, etc.), in response to a planogram (of aretail storage container, of a retail shelf, etc.), in response to arealogram (of a retail storage container, of a retail shelf, etc.), inresponse to a state of at least one product (in a retail storagecontainer, on a retail shelf, etc.), in response to supply chaininformation, in response to an action (such as looking at a product,clicking at a touch screen and/or a key, picking a product, returning aproduct, etc.) of a person (such as a customer, a store associate,etc.), in response to information (such as identity of the person, ageof the person, gender of the person, past behavior of the person,sentiment and/or emotional state of the person, etc.) on a person (suchas a customer, a store associate, etc.), and so forth. Some non-limitingexamples of such elements of a user interface may include a clickableelement, an icon, a textual element, a graphical element, a background,and so forth. For example, in response to a first external trigger, step1708 may cause a first user interface element to be displayed on theelectronic visual display, and in response to a second external trigger,step 1708 may cause a second user interface element to be displayed onthe electronic visual display. In another example, in response to afirst external trigger, step 1708 may cause a first user interfaceelement to be displayed on the electronic visual display, and inresponse to a second external trigger, step 1708 may forgo and/orwithhold causing the display of the first user interface element. Forexample, in response to a first actual inventory (in the retail storagecontainer, on the shelf, etc.), step 1708 may cause a first userinterface element to be displayed on the electronic visual display, andin response to a second actual inventory, step 1708 may cause a seconduser interface element to be displayed on the electronic visual display.In another example, in response to a first actual inventory (in theretail storage container, on the shelf, etc.), step 1708 may cause afirst user interface element to be displayed on the electronic visualdisplay, and in response to a second actual inventory, step 1708 mayforgo and/or withhold causing the display of the first user interfaceelement. For example, in response to a first planogram (of the retailstorage container, of the shelf, etc.), step 1708 may cause a first userinterface element to be displayed on the electronic visual display, andin response to a second planogram, step 1708 may cause a second userinterface element to be displayed on the electronic visual display. Inanother example, in response to a first planogram (of the retail storagecontainer, of the shelf, etc.), step 1708 may cause a first userinterface element to be displayed on the electronic visual display, andin response to a second planogram, step 1708 may forgo and/or withholdcausing the display of the first user interface element. For example, inresponse to a first realogram (of the retail storage container, of theshelf, etc.), step 1708 may cause a first user interface element to bedisplayed on the electronic visual display, and in response to a secondrealogram, step 1708 may cause a second user interface element to bedisplayed on the electronic visual display. In another example, inresponse to a first realogram (of the retail storage container, of theshelf, etc.), step 1708 may cause a first user interface element to bedisplayed on the electronic visual display, and in response to a secondrealogram, step 1708 may forgo and/or withhold causing the display ofthe first user interface element. For example, in response to a firststate of the at least one product (in the retail storage container, onthe shelf, etc.), step 1708 may cause a first user interface element tobe displayed on the electronic visual display, and in response to asecond state of the at least one product, step 1708 may cause a seconduser interface element to be displayed on the electronic visual display.In another example, in response to a first state of the at least oneproduct (in the retail storage container, on the shelf, etc.), step 1708may cause a first user interface element to be displayed on theelectronic visual display, and in response to a second state of the atleast one product, step 1708 may forgo and/or withhold causing thedisplay of the first user interface element. For example, in response tofirst supply chain information, step 1708 may cause a first userinterface element to be displayed on the electronic visual display, andin response to second supply chain information, step 1708 may cause asecond user interface element to be displayed on the electronic visualdisplay. In another example, in response to first supply chaininformation, step 1708 may cause a first user interface element to bedisplayed on the electronic visual display, and in response to secondsupply chain information, step 1708 may forgo and/or withhold causingthe display of the first user interface element. For example, inresponse to a first action (such as looking at a product, clicking at atouch screen and/or a key, picking a product, returning a product, etc.)of a person (such as a customer, a store associate, etc.), step 1708 maycause a first user interface element to be displayed on the electronicvisual display, and in response to a second action of the person, step1708 may cause a second user interface element to be displayed on theelectronic visual display. In another example, in response to a firstaction (such as looking at a product, clicking at a touch screen and/ora key, picking a product, returning a product, etc.) of a person (suchas a customer, a store associate, etc.), step 1708 may cause a firstuser interface element to be displayed on the electronic visual display,and in response to a second action of the person, step 1708 may forgoand/or withhold causing the display of the first user interface element.For example, in response to first information (such as identity, age,gender, past behavior, sentiment and/or emotional state, etc.) on aperson (such as a customer, a store associate, etc.), step 1708 maycause a first user interface element to be displayed on the electronicvisual display, and in response to second information on the person,step 1708 may cause a second user interface element to be displayed onthe electronic visual display. In another example, in response to firstinformation (such as identity, age, gender, past behavior, sentimentand/or emotional state, etc.) of a person (such as a customer, a storeassociate, etc.), step 1708 may cause a first user interface element tobe displayed on the electronic visual display, and in response to secondinformation on the person, step 1708 may forgo and/or withhold causingthe display of the first user interface element. The second userinterface element may differ from the first user interface element. Insome examples, the electronic visual display may be a touch screen, andclicking on the user interface may cause the electronic visual displayto display additional information, may cause transmission of informationto an external system, may trigger a response to the user, and so forth.

In some embodiments, information related to products may be displayed onan electronic visual display (for example as in FIG. 14D, FIG. 14E, FIG.15B, FIG. 15C, FIG. 15E, FIG. 15F, FIG. 15G, FIG. 16B, FIG. 16C and FIG.16D), for example about products associated with a retail storagecontainer (such as products in the retail storage container, productmissing from the retail storage container, and so forth) and/or with aretail shelf (such as products on the shelf, product missing from theshelf, and so forth). In some examples, the displayed informationrelated to products may include images of the products, prices of theproducts, quantity of the products (for example in the retail storagecontainer, on the retail shelf, and so forth), information aboutingredients of the products (such as ‘contains gluten’, ‘gluten free’,list of allergens, calories, fats, sugars, and so forth), Kosherinformation, brand information related to the products, and so forth. Insome examples, step 1708 may select and/or modify the informationrelated to products displayed on the electronic visual display, forexample in response to external triggers, in response to actualinventory (in a retail storage container, on a retail shelf, etc.), inresponse to a planogram (of a retail storage container, of a retailshelf, etc.), in response to a realogram (of a retail storage container,of a retail shelf, etc.), in response to a state of at least one product(in a retail storage container, on a retail shelf, etc.), in response tosupply chain information, in response to an action (such as looking at aproduct, clicking at a touch screen and/or a key, picking a product,returning a product, etc.) of a person (such as a customer, a storeassociate, etc.), in response to information (such as identity of theperson, age of the person, gender of the person, past behavior of theperson, sentiment and/or emotional state of the person, etc.) on aperson (such as a customer, a store associate, etc.), and so forth. Forexample, in response to a first external trigger, step 1708 may causefirst information related to products to be displayed on the electronicvisual display, and in response to a second external trigger, step 1708may cause second information related to products to be displayed on theelectronic visual display. In another example, in response to a firstexternal trigger, step 1708 may cause first information related toproducts to be displayed on the electronic visual display, and inresponse to a second external trigger, step 1708 may forgo and/orwithhold causing the display of the first information related toproducts. For example, in response to a first actual inventory (in theretail storage container, on the shelf, etc.), step 1708 may cause firstinformation related to products to be displayed on the electronic visualdisplay, and in response to a second actual inventory, step 1708 maycause second information related to products to be displayed on theelectronic visual display. In another example, in response to a firstactual inventory (in the retail storage container, on the shelf, etc.),step 1708 may cause first information related to products to bedisplayed on the electronic visual display, and in response to a secondactual inventory, step 1708 may forgo and/or withhold causing thedisplay of the first information related to products. For example, inresponse to a first planogram (of the retail storage container, of theshelf, etc.), step 1708 may cause first information related to productsto be displayed on the electronic visual display, and in response to asecond planogram, step 1708 may cause second information related toproducts to be displayed on the electronic visual display. In anotherexample, in response to a first planogram (of the retail storagecontainer, of the shelf, etc.), step 1708 may cause first informationrelated to products to be displayed on the electronic visual display,and in response to a second planogram, step 1708 may forgo and/orwithhold causing the display of the first information related toproducts. For example, in response to a first realogram (of the retailstorage container, of the shelf, etc.), step 1708 may cause firstinformation related to products to be displayed on the electronic visualdisplay, and in response to a second realogram, step 1708 may causesecond information related to products to be displayed on the electronicvisual display. In another example, in response to a first realogram (ofthe retail storage container, of the shelf, etc.), step 1708 may causefirst information related to products to be displayed on the electronicvisual display, and in response to a second realogram, step 1708 mayforgo and/or withhold causing the display of the first informationrelated to products. For example, in response to a first state of the atleast one product (in the retail storage container, on the shelf, etc.),step 1708 may cause first information related to products to bedisplayed on the electronic visual display, and in response to a secondstate of the at least one product, step 1708 may cause secondinformation related to products to be displayed on the electronic visualdisplay. In another example, in response to a first state of the atleast one product (in the retail storage container, on the shelf, etc.),step 1708 may cause first information related to products to bedisplayed on the electronic visual display, and in response to a secondstate of the at least one product, step 1708 may forgo and/or withholdcausing the display of the first information related to products. Forexample, in response to first supply chain information, step 1708 maycause first information related to products to be displayed on theelectronic visual display, and in response to second supply chaininformation, step 1708 may cause second information related to productsto be displayed on the electronic visual display. In another example, inresponse to first supply chain information, step 1708 may cause firstinformation related to products to be displayed on the electronic visualdisplay, and in response to second supply chain information, step 1708may forgo and/or withhold causing the display of the first informationrelated to products. For example, in response to a first action (such aslooking at a product, clicking at a touch screen and/or a key, picking aproduct, returning a product, etc.) of a person (such as a customer, astore associate, etc.), step 1708 may cause first information related toproducts to be displayed on the electronic visual display, and inresponse to a second action of the person, step 1708 may cause secondinformation related to products to be displayed on the electronic visualdisplay. In another example, in response to a first action (such aslooking at a product, clicking at a touch screen and/or a key, picking aproduct, returning a product, etc.) of a person (such as a customer, astore associate, etc.), step 1708 may cause first information related toproducts to be displayed on the electronic visual display, and inresponse to a second action of the person, step 1708 may forgo and/orwithhold causing the display of the first information related toproducts. For example, in response to first information (such asidentity, age, gender, past behavior, sentiment and/or emotional state,etc.) on a person (such as a customer, a store associate, etc.), step1708 may cause first information related to products to be displayed onthe electronic visual display, and in response to second information onthe person, step 1708 may cause second information related to productsto be displayed on the electronic visual display. In another example, inresponse to first information (such as identity, age, gender, pastbehavior, sentiment and/or emotional state, etc.) of a person (such as acustomer, a store associate, etc.), step 1708 may cause firstinformation related to products to be displayed on the electronic visualdisplay, and in response to second information on the person, step 1708may forgo and/or withhold causing the display of the first informationrelated to products. The second information related to products maydiffer from the first information related to products. In some examples,the electronic visual display may be a touch screen, and clicking on theinformation related to products may cause the electronic visual displayto display additional information, may cause transmission of informationto an external system, and so forth.

In some examples, step 1708 may present information related to availableproducts (for example, available in the retail storage container,available on the retail shelf, etc.) using first display parameters(such as color scheme, size, location, fonts, motion pattern on theelectronic visual display, presentation time, etc.), and may presentinformation related to missing products (for example, missing from theretail storage container, missing from the retail shelf, missingaccording to a planogram, missing according to a realogram, missing incomparison to past inventory, missing in comparison to a shelf label,etc.) using second display parameters. For example, in FIG. 14D and inFIG. 15E and in FIG. 16C, step 1708 may use such display parameters tocontrol the color scheme, and in FIG. 14E and in FIG. 15F and in FIG.16B, may use such display parameters to control the display size and/orthe display location on the electronic visual display, and so forth. Inanother example, such display parameters may control a motion of theinformation related to the products in an animation presented on theelectronic visual display. In yet another example, such displayparameters may control fonts used to display the information. In anadditional example, such display parameters may control the presentationtime of the information.

In some embodiments, step 1708 may use display parameters to presentinformation (for example, to present promotional information, to presentone or more instructions for store associates, to present user interfaceitems, to present information related to products, and so forth). Insome examples, step 1708 may select and/or modify the display parametersin response to external triggers, in response to actual inventory (in aretail storage container, on a retail shelf, etc.), in response to aplanogram (of a retail storage container, of a retail shelf, etc.), inresponse to a realogram (of a retail storage container, of a retailshelf, etc.), in response to a state of at least one product (in aretail storage container, on a retail shelf, etc.), in response tosupply chain information, in response to an action (such as looking at aproduct, clicking at a touch screen and/or a key, picking a product,returning a product, etc.) of a person (such as a customer, a storeassociate, etc.), in response to information (such as identity of theperson, age of the person, gender of the person, past behavior of theperson, sentiment and/or emotional state of the person, etc.) on aperson (such as a customer, a store associate, etc.), and so forth. Somenon-limiting examples of such display parameters may include colorscheme of a displayed item, texture of a displayed item, size of adisplayed item, display location on the electronic visual display of adisplayed item, fonts, motion pattern on the electronic visual displayof a displayed item, presentation time for an item, and so forth. Forexample, in response to a first external trigger, step 1708 may selectfirst display parameters, and in response to a second external trigger,step 1708 may select second display parameters. For example, in responseto a first actual inventory (in the retail storage container, on theshelf, etc.), step 1708 may select first display parameters, and inresponse to a second actual inventory, step 1708 may select seconddisplay parameters. For example, in response to a first planogram (ofthe retail storage container, of the shelf, etc.), step 1708 may selectfirst display parameters, and in response to a second planogram, step1708 may select second display parameters. For example, in response to afirst realogram (of the retail storage container, of the shelf, etc.),step 1708 may select first display parameters, and in response to asecond realogram, step 1708 may select second display parameters. Forexample, in response to a first state of the at least one product (inthe retail storage container, on the shelf, etc.), step 1708 may selectfirst display parameters, and in response to a second state of the atleast one product, step 1708 may select second display parameters. Forexample, in response to first supply chain information, step 1708 mayselect first display parameters, and in response to second supply chaininformation, step 1708 may select second display parameters. Forexample, in response to a first action (such as looking at a product,clicking at a touch screen and/or a key, picking a product, returning aproduct, etc.) of a person (such as a customer, a store associate,etc.), step 1708 may select first display parameters, and in response toa second action of the person, step 1708 may select second displayparameters. For example, in response to first information (such asidentity, age, gender, past behavior, sentiment and/or emotional state,etc.) on a person (such as a customer, a store associate, etc.), step1708 may select first display parameters, and in response to secondinformation on the person, step 1708 may select second displayparameters. The second display parameters may differ from the firstdisplay parameters. In some examples, the electronic visual display maybe a touch screen (for example as described above), and clicking on aportion of the touch screen may cause step 1708 to select differentdisplay parameters.

In some examples, in response to first display parameters, step 1708 maypresent a first visual representation of a particular information, andin response to second display parameters, step 1708 may present a secondvisual representation of the particular information. The second visualrepresentation may differ from the first visual representation, forexample in font, in size, in orientation, in color scheme, in texture,in visual content, in location, in motion pattern, and so forth. Forexample, in response to first display parameters, step 1708 may use afirst font to present a visual representation of the particularinformation, and in response to second display parameters, step 1708 mayuse a second font to present a visual representation of the particularinformation, the second font may differ from the first font. In anotherexample, in response to first display parameters, step 1708 may presenta visual representation of the particular information of a first size,and in response to second display parameters, step 1708 may present avisual representation of the particular information of a second size,the second size may differ from the first size. In yet another example,in response to first display parameters, step 1708 may present a visualrepresentation of the particular information at a first spatialorientation, and in response to second display parameters, step 1708 maypresent a visual representation of the particular information of asecond spatial orientation, the second spatial orientation may differfrom the first spatial orientation. In an additional example, inresponse to first display parameters, step 1708 may use a first colorscheme to present a visual representation of the particular information,and in response to second display parameters, step 1708 may use a secondcolor scheme to present a visual representation of the particularinformation, the second color scheme may differ from the first colorscheme. In another example, in response to first display parameters,step 1708 may present a visual representation of the particularinformation with a first texture, and in response to second displayparameters, step 1708 may present a visual representation of theparticular information with a second texture, the second texture maydiffer from the first texture. In yet another example, in response tofirst display parameters, step 1708 may present a visual representationof the particular information at a first location, and in response tosecond display parameters, step 1708 may present a visual representationof the particular information at a second location, the second locationmay differ from the first location. In an additional example, inresponse to first display parameters, step 1708 may present a visualrepresentation of the particular information moving at a first motionpattern, and in response to second display parameters, step 1708 maypresent a visual representation of the particular information moving ata second motion pattern, the second motion pattern may differ from thefirst motion pattern.

In some examples, an electronic visual display (such as the electronicvisual display of method 1700, of method 1900, of method 2000, of method2100, of method 2200, etc.) may be part of a personal device of a storeassociate, may be part of a personal device of a customer, may beconnected to a shelf in the retail store, may be connected to a door ofa retail storage container in the retail store, and so forth.

A hinged door for a retail storage container with an electronic visualdisplay in the internal part of the door (the part that faces theinternal side of the retail storage container when the door is closed)may enable providence of information to a person (such as a customer, astore associate, etc.) standing in front of the retail storage containerwith the door open. The provided information may be used to drive highersales, to improve customers' experience, and to enhance in-storeexecution.

In some embodiments, a door (such as a hinged door) for a retail storagecontainer is provided. In some examples, the door may comprise at leasta first part that is configured to face customers when the door isclosed and a second part configured to face the internal side of theretail storage container when the door is closed. Some examples of suchdoor may include doors 1300, 1320 and 1340. In one example, the firstpart may comprise at least part of side 1312, and the second part maycomprise at least part of side 1310. In one example, the first part ofthe door may comprise electronic visual display 1322, and the secondpart of the door may comprise electronic visual display 1324. In someexamples, the second part of the door may comprise at least anelectronic visual display configured to display information (such aselectronic visual display 1306, electronic visual display 1324 andelectronic visual display 1342), and at least part of the electronicvisual display may be configured to be visible to the customers at leastwhen the door is open at a selected angle. For example, the at leastpart of the electronic visual display may be configured to be hiddenfrom the customers when the door is closed. Some non-limiting examplesof such retail storage container may include a cooler (such as areach-in cooler, walk-in cooler, display cooler, countertop cooler,under-counter cooler, worktop cooler, chest cooler, merchandisingcooler, etc.), a refrigerator unit (such as a reach-in refrigerator,display refrigerator, walk-in refrigerator, countertop refrigerator,under-counter refrigerator, worktop refrigerator, chest refrigerator,merchandising refrigerator, etc.), a freezer (such as a reach-infreezer, walk-in freezer, display freezer, countertop freezer,under-counter freezer, worktop freezer, chest freezer, merchandisingfreezer, etc.), a closet, enclosed storage unit with a door, shelvingunit with a door, or any other unit configured to include at least onedoor and is configured to hold one or more products for sale in a retailestablishment.

In one example, the information displayed by the electronic visualdisplay may include promotional information. In another example, theinformation displayed by the electronic visual display may includeinstructions for a store associate. In yet another example, theinformation displayed by the electronic visual display may includeelements of a user interface. In an additional example, the informationdisplayed by the electronic visual display may include informationrelated to products. In some examples, the information displayed by theelectronic visual display may be controlled using one or more of methods1700, 1800, 1900, 2000, 2100 and 2200, or using one or more of the stepsof the above identified methods.

In some examples, the information displayed by the electronic visualdisplay may be based on a person facing the retail storage containerand/or on a person in a vicinity of the retail storage container. In oneexample, in response to a first person, first information may bepresented by the electronic visual display, and in response to a secondperson, second information may be presented by the electronic visualdisplay, the second information may differ from the first information.In another example, in response to a first person, first information maybe presented by the electronic visual display, and in response to asecond person, presenting the first information by the electronic visualdisplay may be withheld. In one example, a determination of whether theperson is a customer may be made (for example as described below), andthe information displayed by the electronic visual display may be basedon the determination of whether the person is a customer. In anotherexample, a determination of whether the person is a store associate maybe made (for example as described below), and the information displayedby the electronic visual display may be based on the determination ofwhether the person is a store associate. In yet another example, adetermination of whether the person belongs to a particular group ofpeople (such as a particular group of store associates, a particulargroup of customers, etc.) may be made (for example as described below),and the information displayed by the electronic visual display may bebased on the determination of whether the person belongs to theparticular group of people. In an additional example, demographicinformation of the person (such as age, gender, a socio-economic group,etc.) may be determined (for example as described below), and theinformation displayed by the electronic visual display may be based onthe determined demographic information (for example, based on thedetermined age, based on the determined gender, based on the determinedsocio-economic group, and so forth). In another example, past behaviorof the person may be determined (for example, one or more productspicked by the person may be determined, a trajectory of the person maybe determined, purchase history of the person may be determined, etc.),and the information displayed by the electronic visual display may bebased on the determined past behavior of the person (for example, basedon the one or more products picked by the person, based on the atrajectory of the person, based on purchase history of the person may,and so forth). In an additional example, an identity of the person maybe determined (for example as described below), and the informationdisplayed by the electronic visual display may be based on thedetermined identity of the person.

In some examples, the information displayed by the electronic visualdisplay may be based on data related to products stored in the retailstorage container. In one example, in response to first plurality ofproducts stored in the retail storage container, first information maybe presented by the electronic visual display, and in response to secondplurality of products stored in the retail storage container, secondinformation may be presented by the electronic visual display, thesecond information may differ from the first information. In anotherexample, in response to first plurality of products stored in the retailstorage container, first information may be presented by the electronicvisual display, and in response to second plurality of products storedin the retail storage container, presenting the first information by theelectronic visual display may be withheld. In one example, an inventoryof products stored in the retail storage container may be determined(for example as described below), and the information displayed by theelectronic visual display may be based on the determined inventory ofproducts stored in the retail storage container. In another example, atype of a product stored in the retail storage container may bedetermined (for example as described below), and the informationdisplayed by the electronic visual display may be based on thedetermined type of the product stored in the retail storage container.In yet another example, data related to facings of products stored inthe retail storage container may be determined (for example as describedbelow), and the information displayed by the electronic visual displaymay be based on the determined data related to the facings of theproducts stored in the retail storage container.

In some examples, the information displayed by the electronic visualdisplay may be based on a label positioned in the retail storagecontainer. In one example, in response to first label positioned in theretail storage container, first information may be presented by theelectronic visual display, and in response to second label positioned inthe retail storage container, second information may be presented by theelectronic visual display, the second information may differ from thefirst information. In another example, in response to first labelpositioned in the retail storage container, first information may bepresented by the electronic visual display, and in response to secondlabel positioned in the retail storage container, presenting the firstinformation by the electronic visual display may be withheld. In oneexample, a price displayed on the label may be determined, for exampleby analyzing an image of the label using OCR algorithms, and theinformation displayed by the electronic visual display may be based onthe determined price displayed on the label. In another example, aproduct associated with the label may be determined (for example asdescribed below), and the information displayed by the electronic visualdisplay may be based on the determined product associated with thelabel. In yet another example, a visual code (such as a barcode, a QRcode, a serial number, etc.) displayed on the label may be identified,for example by analyzing an image of the label using a visual codeidentification algorithm, and the information displayed by theelectronic visual display may be based on the identified visual codedisplayed on the label. In an additional example, a product depicted onthe label may be identified, for example by analyzing an image of thelabel using a visual object recognition algorithm, and the informationdisplayed by the electronic visual display may be based on theidentified product depicted on the label.

In some examples, the retail storage container may comprise an imagesensor, such as an image sensor positioned within the retail storagecontainer, and the second part of the door may further comprise a mirrorconfigured to reflect towards the image sensor an image of at least aportion of an internal part of the retail storage container. In oneexample, the information displayed by the electronic visual display maybe based on an analysis of the image reflected by the mirror anddigitally captured using the image sensor, for example based on productsand/or labels and/or textual information visible in the image. In oneexample, the image sensor may be configured to capture an image of aperson facing the retail storage container when the door is open, andthe information displayed by the electronic visual display may be basedon an analysis of the image of the person facing the retail storagecontainer. In one example, an indication that the door is closed may bereceived (for example, from a sensor connected to the door, from asensor connected to the retail storage container, from an analysis ofone or more images, etc.), and in response to the received indication,the image sensor may be caused to capture at least one image. In someexamples, the retail storage container may comprise a shelf, and themirror may be configured to reflect towards the image sensor an image ofat least part of the shelf and of an area above the shelf. In oneexample, the mirror may be configured to reflect towards the imagesensor an image of at least part of the shelf, an area above the shelf,and an area below the shelf In another example, the mirror may beconfigured to reflect towards the image sensor an image of at least partof the shelf, and at least part of one or more products positioned onthe shelf. In yet another example, the mirror may be configured toreflect towards the image sensor an image of at least part of the shelf,at least part of one or more products positioned on the shelf, and atleast part of one or more products positioned below the shelf. In anadditional example, the mirror may be configured to reflect towards theimage sensor an image of at least part of a label attached to the shelf.

In some examples, the second part of the door may further comprise animage sensor configured to capture at least one image of at least aportion of an internal part of the retail storage container. In oneexample, the information displayed by the electronic visual display maybe based on an analysis of an analysis of the at least one image, forexample based on products and/or labels and/or textual informationvisible in the at least one image. In one example, the image sensor maybe configured to capture an image of a person facing the retail storagecontainer when the door is open, and the information displayed by theelectronic visual display may be based on an analysis of the image ofthe person facing the retail storage container. In one example, anindication that the door is closed may be received (for example, from asensor connected to the door, from a sensor connected to the retailstorage container, from an analysis of one or more images, etc.), and inresponse to the received indication, the image sensor may be caused tocapture the at least one image. In some examples, the retail storagecontainer may comprise a shelf, and the image sensor may be configuredto capture an image of at least part of the shelf and/or of an areaabove the shelf. For example, the image sensor may be configured tocapture an image of at least part of the shelf, an area above the shelf,and an area below the shelf. In another example, the image sensor may beconfigured to capture an image of at least part of the shelf, and atleast part of one or more products positioned on the shelf. In yetanother example, the image sensor may be configured to capture an imageof at least part of the shelf, at least part of one or more productspositioned on the shelf, and at least part of one or more productspositioned below the shelf. In an additional example, the image sensormay be configured to capture an image of at least part of a labelattached to the shelf.

In some examples, the retail storage container may comprise a shelf, aplurality of sensors may be positioned on the shelf and may beconfigured to be positioned between the shelf and products positioned onthe shelf (for example as described in relation to FIG. 8A, 8B and 9),and the information displayed by the electronic visual display may bebased on an analysis of data captured using the plurality of sensors(for example as described below in relation to methods 1800, 1900 and2100). In some examples, the retail storage container may comprise ashelf, a plurality of pressure sensors may be positioned on the shelfand may be configured to be positioned between the shelf and productspositioned on the shelf, and the information displayed by the electronicvisual display may be based on an analysis of pressure data capturedusing the plurality of pressure sensors (for example as described belowin relation to methods 1800, 1900 and 2100). In some examples, theretail storage container may comprise a shelf, a plurality of touchsensors may be positioned on the shelf and may be configured to bepositioned between the shelf and products positioned on the shelf, andthe information displayed by the electronic visual display may be basedon an analysis of touch data captured using the plurality of touchsensors (for example as described below in relation to methods 1800,1900 and 2100). In some examples, the retail storage container maycomprise a shelf, a plurality of light sensors may be positioned on theshelf and may be configured to be positioned between the shelf andproducts positioned on the shelf, and the information displayed by theelectronic visual display may be based on an analysis of light datacaptured using the plurality of light sensors (for example as describedbelow in relation to methods 1800, 1900 and 2100). In some examples, theretail storage container may comprise a shelf, and the informationdisplayed by the electronic visual display may be based on an analysisof weight data captured using a weight sensor (for example as describedbelow in relation to methods 1800, 1900 and 2100). For example, theweight sensor may be configured to measure a weight of at least oneproduct placed on the shelf

In some examples, an indication of a state of the door may be received,for example, from a sensor connected to the door, from a sensorconnected to the retail storage container, from an analysis of one ormore images, and so forth. Some non-limiting examples of such possiblestates of the door may include open, closed, partly open, open at aparticular angle, open at an angle that is within a selected range ofangles, partly open to a particular degree, partly open to a degree thatis within a selected range of degrees, and so forth. In one example, inresponse to a first state of the door, the electronic visual display maybe caused to display the information, and in response to a second stateof the door, causing the electronic visual display to display theinformation may be forgone and/or withheld.

In one example, in response to a first state of the door, the electronicvisual display may be caused to display first information, and inresponse to a second state of the door, the electronic visual displaymay be caused to display second information, the second information maydiffer from the first information. In one example, an indication ofwhether the door is open may be received, in response to an indicationthat the door is open, the electronic visual display may be caused todisplay the information, and in response to an indication that the dooris closed, causing the electronic visual display to display theinformation may be forgone and/or withheld. In one example, anindication of a degree of openness of the door may be received, inresponse to a first degree of openness of the door, the electronicvisual display may be caused to display the information, and in responseto a second degree of openness of the door, causing the electronicvisual display to display the information may be forgone and/orwithheld. In one example, an indication of whether the door is open maybe received, and an adjustment to a power scheme of the electronicvisual display may be caused based on the received indication. In oneexample, an indication of whether the door is open may be received, inresponse to an indication that the door is open, the electronic visualdisplay may be caused to turn on, and in response to an indication thatthe door is closed, the electronic visual display may be caused to turnoff. In one example, an indication of a degree of openness of the doormay be received, in response to a first degree of openness of the door,the electronic visual display may be caused to turn on, and in responseto a second degree of openness of the door, the electronic visualdisplay may be caused to turn off.

In some examples, different determinations on a person may be made. Forexample a determination of whether the person is a customer may be made,a determination of whether the person is a store associate may be made,a determination of whether the person belongs to a particular group ofpeople (such as a particular group of store associates, a particulargroup of customers, etc.) may be made, a determination of demographicinformation of the person (such as age, gender, a socio-economic group,etc.) may be made, a determination of past behavior of the person may bemade (for example, one or more products picked by the person may bedetermined, a trajectory of the person may be determined, purchasehistory of the person may be determined, etc.), a determination of anidentity of a person may be made, and so forth. In some examples, animage of the person may be analyzed, for example using a facerecognition algorithm, to access a database comprising information ondifferent people, and the accessed information may be used to make anyof the above determinations on the person. In one example, such imagemay be captured from an environment of the retail store using an imagesensor. In some examples, a wireless signal from a personal device ofthe person may be received, the wireless signal may include a uniqueidentifier (such as a MAC address, a loyalty card number, an employeenumber, etc.) corresponding to the personal device and/or to the person,and the unique identifier may be used to access a database including arecord with information related to the person, and the informationrelated to the person may be used to make any of the abovedeterminations on the person. In an additional information, a trackingalgorithms (such as a visual tracking algorithm, a wireless signaltracking algorithm, etc.) may be used to determine past behavior of theperson, such as locations within the retail store that the personvisited, frequent, stopped by, and so forth. In yet another example,image analysis algorithm to determine sentiment and/or emotional stateof the person from an image of the person. In an additional example, awireless signal from a personal device of the person may be received,the wireless signal may include a record with information related to theperson, and the information related to the person may be used to makeany of the above determinations on the person. In an additional example,the different determinations on a person may be made using step 1706.

In some examples, information related to a label may be determined, suchas a product related to the shelf label, a price associated with theshelf label, a brand associated with the shelf label, and so forth. Forexample, an image of the label may be analyzed using OCR to recognizetext appearing on the label, and the text may include the information(for example, the product name, the brand name, the price, and soforth). In another example, an image of the label may be analyzed usinga product recognition algorithm to identify a product from a depictionof at least part of the product on the label, and the identity of theproduct may be used to determine the product name, the correspondingbrand name, the corresponding price, and so forth. In yet anotherexample, an image of the label may be analyzed using a logo recognitionalgorithm to identify a brand from a logo appearing on the label, andthe identified brand may be used to determine the brand name. In anadditional example, an image of the label may be analyzed using a visualcode reading algorithm to read a visual code appearing on the label(such as a barcode, a QR code, a serial number, etc.), and the read codemay be used to access a record in a database including the informationrelated to the label.

A door for a retail storage container with a transparent electronicvisual display may enable providence of visual information to a person(such as a customer, a store associate, etc.) standing in front of theretail storage container. The provided information may be used to drivehigher sales, to improve customers' experience, and to enhance in-storeexecution. The presentation of the information on selected regions ofthe transparent electronic visual display may create an overlay ofinformation over the products and/or shelves in the retail storagecontainer that are visible through the transparent electronic visualdisplay, therefore visually associating the provided information withthe overlaid products and/or shelves.

FIG. 18 provides a flowchart of an exemplary method 1800 for controllinginformation displayed on a transparent electronic visual display that ispart of a door for a retail storage container, consistent with thepresent disclosure. In this example, method 1800 for controllinginformation displayed on a transparent electronic display that is partof a door for a retail storage container may comprise: receiving anindication of at least one position associated with a first product typein the retail storage container (step 1802); receiving an indication ofat least one position associated with a second product type in theretail storage container (step 1804); using the indication of the atleast one position associated with the first product type to select afirst region of the transparent electronic display (step 1806); usingthe indication of the at least one position associated with the secondproduct type to select a second region of the transparent electronicdisplay (step 1808); displaying visual information related to the firstproduct type on the first region of the transparent electronic display(step 1810); and displaying visual information related to the secondproduct type on the second region of the transparent electronic display(step 1812). In one example, steps 1804, 1808 and 1812 may be omittedfrom method 1800.

In some examples, step 1802 may comprise receiving an indication of atleast one position associated with a first product type in the retailstorage container, and step 1804 may comprise receiving an indication ofat least one position associated with a second product type in theretail storage container. The second product type may differ from thefirst product type. Some non-limiting examples of such indication of aposition of a product may include any combination of one or more of aheight indication, a vertical position indication, a horizontal positionindication, a shelf indication, an indication of a position on theshelf, and so forth. For example, such indications of at least oneposition associated with a particular product type may be read frommemory (for example, from memory 226 or from memory 1226), may bereceived from an external system (for example, using network interface206), may be determined by analyzing images of the retail storagecontainer (for example as described herein), may be determined byanalyzing data captured using sensors positioned between a shelf in theretail storage container and products placed on the shelf (for exampleas described herein), and so forth.

In one example, the at least one position associated with the firstproduct type may include a position of the first product type in aplanogram, and/or the at least one position associated with the secondproduct type may include a position of the second product type in theplanogram. Further, in one example, the indication of the at least oneposition associated with the first product type received by step 1802and the indication of the at least one position associated with thesecond product type received by step 1804 may be based on an analysis ofthe planogram.

In one example, the at least one position associated with the firstproduct type may include an actual position of products of the firstproduct type in the retail storage container, and/or the at least oneposition associated with the second product type may include an actualposition of products of the second product type in the retail storagecontainer. For example, the actual position of the products of thedifferent product types may be determined by analyzing images of theproducts, by analyzing data captured using sensors positioned between ashelf in the retail storage container and products placed on the shelf,and so forth, for example as described herein.

In one example, the at least one position associated with the firstproduct type may include a position of a label corresponding to thefirst product type in the retail storage container, and/or the at leastone position associated with the second product type may include aposition of a label corresponding to the second product type in theretail storage container. For example, the position of the labelscorresponding to the product types may be determined by analyzing imagesof the labels to identify a location of the labels and/or correspondenceof the labels to different product types, for example based on textualinformation presented on the labels (for example using OCR algorithms),based on visual code presented on the label (for example using visualcode recognition algorithms), based on an image of the product (forexample using product recognition algorithms), and so forth.

In one example, the at least one position associated with the firstproduct type may include a position of an empty space dedicated to thefirst product type in the retail storage container, and/or the at leastone position associated with the second product type may include aposition of an empty space dedicated to the second product type in theretail storage container. For example, empty space dedicated to aproduct type may be identified by comparing the empty spaces in theretail storage container to a planogram and/or to a realogram. In oneexample, the empty spaces in the retail storage container may beidentified by analyzing images of the retail storage container using aproduct detection algorithm to identify regions of the retail storagecontainer that hold no products, by analyzing data captured usingsensors (such as pressure sensors, touch sensors, light sensors, weightsensors, etc.) positioned on a shelf in the retail storage container,and so forth, for example as described herein.

In one example, the at least one position associated with the firstproduct type may include a position at which products of the firstproduct type were previously placed in the retail storage container andat which products of the first product type are not currently placed,and/or the at least one position associated with the second product typemay include a position at which products of the second product type werepreviously placed in the retail storage container and at which productsof the second product type are not currently placed. For example, aposition at which products of a particular product type were previouslyplaced in the retail storage container and at which products of theparticular product type are not currently placed may be identified byanalyzing images from the two point in time using product detectionand/or recognition algorithms, by analyzing patterns in data capturedusing sensors (such as pressure sensors, touch sensors, light sensors,weight sensors, etc.) positioned between a shelf in the retail storagecontainer and products placed on the shelf, and so forth, for example asdescribed herein.

In one example, the indication of the at least one position associatedwith the first product type received by step 1802 may be based on ananalysis of at least one image of products placed in the retail storagecontainer, and/or the indication of the at least one position associatedwith the second product type received by step 1804 may be based on ananalysis of the at least one image of products placed in the retailstorage container. For example, product detection and/or recognitionalgorithms may be used to analyze the at least one image and identify topositions of products of different product types in the retail storagecontainer.

In some examples, the retail storage container may comprise a shelf, aplurality of sensors may be positioned on the shelf and configured to bepositioned between the shelf and products positioned on the shelf (forexample as described in relation to FIG. 8A, 8B and 9), the indicationof the at least one position associated with the first product typereceived by step 1802 may be based on an analysis of data captured usingthe plurality of sensors (for example as described in relation to FIG.10A and 10B), and/or the indication of the at least one positionassociated with the second product type received by step 1804 may bebased on an analysis of data captured using the plurality of sensors(for example as described in relation to FIG. 10A and 10B). In oneexample, the retail storage container may comprise a shelf, a pluralityof pressure sensors may be positioned on the shelf and may be configuredto be positioned between the shelf and products positioned on the shelf,and the indication of the at least one position associated with thefirst product type received by step 1802 and/or the indication of the atleast one position associated with the second product type received bystep 1804 may be based on an analysis of pressure data captured usingthe plurality of pressure sensors. In some examples, the retail storagecontainer may comprise a shelf, a plurality of touch sensors may bepositioned on the shelf and may be configured to be positioned betweenthe shelf and products positioned on the shelf, and the indication ofthe at least one position associated with the first product typereceived by step 1802 and/or the indication of the at least one positionassociated with the second product type received by step 1804 may bebased on an analysis of touch data captured using the plurality of touchsensors. In some examples, the retail storage container may comprise ashelf, a plurality of light sensors may be positioned on the shelf andmay be configured to be positioned between the shelf and productspositioned on the shelf, and the indication of the at least one positionassociated with the first product type received by step 1802 and/or theindication of the at least one position associated with the secondproduct type received by step 1804 may be based on an analysis of lightdata captured using the plurality of light sensors. In some examples,the retail storage container may comprise a shelf, and the indication ofthe at least one position associated with the first product typereceived by step 1802 may be based on an analysis of weight datacaptured using the weight sensor, the weight sensor may be configured tomeasure a weight of at least one product placed on at least part of theshelf

In some examples, step 1806 may comprise using the indication of the atleast one position associated with the first product type to select afirst region of the transparent electronic display, and step 1808 maycomprise using the indication of the at least one position associatedwith the second product type to select a second region of thetransparent electronic display. The second region may differ from thefirst region. For example, the selection of the first region of thetransparent electronic display by step 1806 may be configured to causeat least part of the displayed visual information related to the firstproduct type to appear over at least part of the at least one positionassociated with the first product type when viewed from a particularviewing point, and the selection of the second region of the transparentelectronic display by step 1808 may be configured to cause at least partof the displayed visual information related to the second product typeto appear over at least part of the at least one position associatedwith the second product type when viewed from the particular viewingpoint. For example, geometrical analysis may be used to select a regionof the transparent electronic display that is on a straight lineconnecting the particular viewing point and the corresponding at leastone position associated with the corresponding product type. In anotherexample, a predefined mapping of positions associated with product typesto regions of the transparent electronic display may be used to selectthe region of the transparent electronic display corresponding to theproduct type based on the indication of the at least one positionassociated with the product type. The predefined mapping may beconfigured to select a region that causes visual information displayedin the selected region to appear over at least part of the at least oneposition associated with the corresponding product type when viewed fromthe particular viewing point.

In some examples, an indication of a state of the door may be received,for example, from a sensor connected to the door, from a sensorconnected to the retail storage container, from an analysis of one ormore images, and so forth. Some non-limiting examples of such possiblestates of the door may include open, closed, partly open, open at aparticular angle, open at an angle that is within a selected range ofangles, partly open to a particular degree, partly open to a degree thatis within a selected range of degrees, and so forth. In some examples,the selection of the first region of the transparent electronic displayby step 1806 and the selection of the second region of the transparentelectronic display by step 1808 may be based on the state of the door.For example, in response to a first received indication of the state ofthe door, step 1806 may select one region as the first region of thetransparent electronic display, and in response to a second receivedindication of the state of the door, step 1806 may select a differentregion as the first region of the transparent electronic display. In oneexample, an indication of whether the door is open may be received, inresponse to an indication that the door is open, step 1806 may selectone region as the first region of the transparent electronic display,and in response to an indication that the door is closed, step 1806 mayselect a different region as the first region of the transparentelectronic display. In one example, an indication of a degree ofopenness of the door may be received, in response to a first degree ofopenness of the door, step 1806 may select one region as the firstregion of the transparent electronic display, and in response to asecond degree of openness of the door, step 1806 may select a differentregion as the first region of the transparent electronic display.

In some examples, the selection of the first region of the transparentelectronic display by step 1806 and the selection of the second regionof the transparent electronic display by step 1808 may be based on aperson facing the retail storage container, for example on a height ofthe person, on a position of a face of the person, on a position of atleast one eye of the person, on an orientation of a face of the person,on a direction of a gaze of the person, and so forth. For example, theparticular viewing point discussed above may be selected based on aheight of the person, on a position of a face of the person, on aposition of at least one eye of the person, on an orientation of a faceof the person, on a direction of a gaze of the person, and so forth. Inone example, in response to one posture of the person facing the retailstorage container, step 1806 may select one region of the transparentelectronic display, and in response to a different posture of the personfacing the retail storage container, step 1806 may select a differentregion of the transparent electronic display.

In some examples, step 1810 may comprise displaying visual informationrelated to the first product type on the first region of the transparentelectronic display, and step 1812 may comprise displaying visualinformation related to the second product type on the second region ofthe transparent electronic display. Some non-limiting examples of suchvisual information related to a product type may include a visualindication of a price corresponding to the product type, a visualindication of a name corresponding to the product type (such as a nameof the product type, a brand name corresponding to the product type, andso forth), a promotion corresponding to the product type, an indicationof a need to restock the product type in the retail storage container,an indication of a need to remove products of the product type from theretail storage container, an indication of a need to collect products ofthe product type from the retail storage container, an indication of aneed to handle a label corresponding to the product type in the retailstorage container, and so forth. For example, the visual informationrelated to the first product type displayed by step 1810 may include aprice corresponding to the first product type, and/or the visualinformation related to the second product type displayed by step 1812may include a price corresponding to the second product type. In anotherexample, the visual information related to the first product typedisplayed by step 1810 may include a name corresponding to the firstproduct type (such as a name of the first product type, a brand namecorresponding to the first product type, and so forth), and/or thevisual information related to the second product type displayed by step1810 may include a name corresponding to the second product type (suchas a name of the second product type, a brand name corresponding to thesecond product type, and so forth). In yet another example, the visualinformation related to the first product type displayed by step 1810 mayinclude a promotion corresponding to the first product type, and/or thevisual information related to the second product type displayed by step1812 may include a promotion corresponding to the second product type.In an additional example, the visual information related to the firstproduct type displayed by step 1810 may include an indication of a needto restock the first product type in the retail storage container,and/or the visual information related to the second product typedisplayed by step 1812 may include an indication of a need to repositionproducts of the first product type in the retail storage container. Inanother example, the visual information related to the first producttype displayed by step 1810 may include an indication of a need toremove products of the first product type from the retail storagecontainer, and/or the visual information related to the second producttype displayed by step 1812 may include an indication of a need toremove products of the second product type from the retail storagecontainer. In yet another example, the visual information related to thefirst product type displayed by step 1810 may include an indication of aneed to collect products of the first product type from the retailstorage container, and/or the visual information related to the secondproduct type displayed by step 1812 may include an indication of a needto collect products of the second product type from the retail storagecontainer. In an additional example, the visual information related tothe first product type displayed by step 1810 may include an indicationof a need to handle a label corresponding to the first product type inthe retail storage container, and/or the visual information related tothe second product type displayed by step 1812 may include an indicationof a need to handle a label corresponding to the second product type inthe retail storage container. In some examples, step 1810 may select thevisual information related to the first product type for display and/orstep 1812 may select the visual information related to the secondproduct type for display using method 1700. In some examples, step 1810may determine whether to display the visual information related to thefirst product type and/or step 1812 may determine whether to display thevisual information related to the second product type using method 1900and/or method 2100. In some examples, step 1810 may select displayparameters for the display of the visual information related to thefirst product type and/or step 1812 may select display parameters forthe display of the visual information related to the second product typeusing method 2000 and/or method 2200.

In some examples, the retail storage container may comprise a shelf, aplurality of sensors may be positioned on the shelf and may beconfigured to be positioned between the shelf and products positioned onthe shelf (for example as described in relation to FIG. 8A, 8B and 9),and the visual information related to the first product type displayedby step 1810 may be based on an analysis of data captured using theplurality of sensors, and/or the visual information related to thesecond product type displayed by step 1812 may be based on the analysisof data captured using the plurality of sensors, for example asdescribed herein. In one example, the data captured using the pluralityof sensors may be analyzed using step 1704. In one example, types ofproducts, positions of products, facings of products, inventory, etc.may be identified by analyzing the data captured using the plurality ofsensors, as described above, and the displayed visual informationrelated to a product type may be based on such identified information,for example as described below. In one example, the retail storagecontainer may comprise a shelf, a plurality of pressure sensors may bepositioned on the shelf and may be configured to be positioned betweenthe shelf and products positioned on the shelf, and the visualinformation related to the first product type displayed by step 1810and/or the visual information related to the second product typedisplayed by step 1812 may be based on an analysis of pressure datacaptured using the plurality of pressure sensors. In one example, theretail storage container may comprise a shelf, a plurality of touchsensors may be positioned on the shelf and may be configured to bepositioned between the shelf and products positioned on the shelf, andthe visual information related to the first product type displayed bystep 1810 and/or the visual information related to the second producttype displayed by step 1812 may be based on an analysis of touch datacaptured using the plurality of touch sensors. In one example, theretail storage container may comprise a shelf, a plurality of lightsensors may be positioned on the shelf and configured to be positionedbetween the shelf and products positioned on the shelf, and the visualinformation related to the first product type displayed by step 1810and/or the visual information related to the second product typedisplayed by step 1812 may be based on an analysis of light datacaptured using the plurality of light sensors. In one example, theretail storage container may comprise a shelf, and the visualinformation related to the first product type displayed by step 1810and/or the visual information related to the second product typedisplayed by step 1812 may be based on an analysis of weight datacaptured using the weight sensor. For example, the weight sensor may beconfigured to measure a weight of at least one product placed on theshelf.

In some examples, the visual information related to the first producttype displayed by step 1810 and/or the visual information related to thesecond product type displayed by step 1812 may be based on an analysisof at least one image of products placed in the retail storagecontainer, for example as described above. In one example, the at leastone image of products placed in the retail storage container may becaptured using an image sensor connected to the retail storagecontainer, using an image sensor connected to a door of the retailstorage container, using a mirror connected to a door of the retailstorage container (as described above), and so forth. In one example,the at least one image of products placed in the retail storagecontainer may be received using step 1702. In one example, the at leastone image of products placed in the retail storage container may beanalyzed using step 1704. In one example, types of products, positionsof products, condition of products, facings of products, inventory, etc.may be identified by analyzing the at least one image, as describedabove, and the displayed visual information related to a product typemay be based on such identified information, for example as describedbelow.

In some examples, the visual information related to the first producttype displayed by step 1810 may be based on an amount of products of thefirst product type placed in the retail storage container and/or thevisual information related to the second product type displayed by step1812 may be based on an amount of products of the second product typeplaced in the retail storage container. For example, in response to afirst amount of products of a particular product type placed in theretail storage container, first visual information related to theparticular product type may be displayed, and in response to a secondamount of products of a particular product type placed in the retailstorage container, second visual information related to the particularproduct type may be displayed, the second visual information may differfrom the first visual information. In another example, in response to afirst amount of products of a particular product type placed in theretail storage container, first visual information related to theparticular product type may be displayed, and in response to a secondamount of products of a particular product type placed in the retailstorage container, displaying the first visual information may beforgone and/or withheld. In one example, an amount of products of thefirst product type in the retail storage container may be obtained (forexample, by analyzing at least one image of the product, by analyzingdata captured using a plurality of sensors positioned between the shelfand products positioned on the shelf, using any of the methods describedherein, etc.), the amount of products of the first product type in theretail storage container may be compared with a selected threshold, inresponse to a first result of the comparison, step 1810 may displayfirst visual information related to the first product type, and inresponse to a second result of the comparison, step 1810 may displaysecond visual information related to the first product type, the secondvisual information may differ from the first visual information.

In some examples, the visual information related to the first producttype displayed by step 1810 may be based on facings of the first producttype in the retail storage container and/or the visual informationrelated to the second product type displayed by step 1812 may be basedon facings of the second product type in the retail storage container.For example, in response to a first facings configuration of aparticular product type in the retail storage container, first visualinformation related to the particular product type may be displayed, andin response to a second facings configuration of the particular producttype in the retail storage container, second visual information relatedto the particular product type may be displayed, the second visualinformation may differ from the first visual information.

In another example, in response to a first facings configuration of aparticular product type in the retail storage container, first visualinformation related to the particular product type may be displayed, andin response to a second facings configuration of the particular producttype in the retail storage container, displaying the first visualinformation may be forgone and/or withheld.

In some examples, the visual information related to the first producttype displayed by step 1810 may be based on an information presented ona label corresponding to the first product type and/or the visualinformation related to the second product type displayed by step 1812may be based on information presented on a label corresponding to thesecond product type. For example, in response to a first informationpresented on a label corresponding to a particular product type, firstvisual information related to the particular product type may bedisplayed, and in response to a second information presented on thelabel corresponding to the particular product type, second visualinformation related to the particular product type may be displayed, thesecond visual information may differ from the first visual information.In another example, in response to a first information presented on alabel corresponding to a particular product type, first visualinformation related to the particular product type may be displayed, andin response to a second information presented on the label correspondingto the particular product type, displaying the first visual informationmay be forgone and/or withheld.

In some examples, the visual information related to the first producttype displayed by step 1810 may be based on a price corresponding to thefirst product type and/or the visual information related to the secondproduct type displayed by step 1812 may be based on a pricecorresponding to the second product type. For example, in response to afirst price corresponding to a particular product type, first visualinformation related to the particular product type may be displayed, andin response to a second price corresponding to the particular producttype, second visual information related to the particular product typemay be displayed, the second visual information may differ from thefirst visual information. In another example, in response to a first aprice corresponding to a particular product type, first visualinformation related to the particular product type may be displayed, andin response to a second price corresponding to the particular producttype, displaying the first visual information may be forgone and/orwithheld.

In some examples, the visual information related to the first producttype displayed by step 1810 may be based on the first region of thetransparent electronic display selected by step 1806 and/or the visualinformation related to the second product type displayed by step 1812may be based on the second region of the transparent electronic displayselected by step 1808. For example, in response to a first selection ofthe first region of the transparent electronic display selected by step1806, first visual information related to the particular product typemay be displayed, and in response to a second selection of the firstregion of the transparent electronic display selected by step 1806,second visual information related to the particular product type may bedisplayed, the second visual information may differ from the firstvisual information. In another example, in response to a first selectionof the first region of the transparent electronic display selected bystep 1806, first visual information related to the particular producttype may be displayed, and in response to a second selection of thefirst region of the transparent electronic display selected by step1806, displaying the first visual information may be forgone and/orwithheld.

In some examples, the visual information related to the first producttype displayed by step 1810 may be based on the at least one positionassociated with the first product type in the retail storage container(for example as indicated by the indication received by step 1802)and/or the visual information related to the second product typedisplayed by step 1812 may be based on the at least one positionassociated with the second product type in the retail storage container(for example as indicated by the indication received by step 1804). Forexample, in response to a first indication of the at least one positionassociated with the first product type in the retail storage containerreceived by step 1802, first visual information related to theparticular product type may be displayed, and in response to a secondindication of the at least one position associated with the firstproduct type in the retail storage container received by step 1802,second visual information related to the particular product type may bedisplayed, the second visual information may differ from the firstvisual information. In another example, in response to a firstindication of the at least one position associated with the firstproduct type in the retail storage container received by step 1802,first visual information related to the particular product type may bedisplayed, and in response to a second indication of the at least oneposition associated with the first product type in the retail storagecontainer received by step 1802, displaying the first visual informationmay be forgone and/or withheld.

In some examples, the visual information related to the first producttype displayed by step 1810 and/or the visual information related to thesecond product type displayed by step 1812 may be based on a personfacing the retail storage container. For example, in response to a firstperson facing the retail storage container, first visual informationrelated to a particular product type may be displayed, and in responseto a second person facing the retail storage container, second visualinformation related to the particular product type may be displayed, thesecond visual information may differ from the first visual information.In another example, in response to a first person facing the retailstorage container, first visual information related to a particularproduct type may be displayed, and in response to a second person facingthe retail storage container, displaying the first visual informationmay be forgone and/or withheld.

Providing selected visual information to a person (such as a customer, astore associate, etc.) may be used to drive higher sales, to improvecustomers' experience, and to enhance in-store execution. Correctselection of the information and correct selection of the visualappearance of the information may help obtaining these objectives.

FIG. 19 provides a flowchart of an exemplary method 1900 for selectingitems for presentation on electronic visual displays in retail stores,consistent with the present disclosure. In this example, method 1900 forselecting items for presentation on electronic visual displays in retailstores may comprise: obtaining a plurality of images of products in aretail store captured using at least one image sensor (step 1902), theplurality of images may comprise at least a first image corresponding toa first point in time and a second image corresponding to a second pointin time, the first point in time is earlier than the second point intime; analyzing the first image to determine whether products of aparticular product type are available at the first point in time (step1904); analyzing the second image to determine whether products of theparticular product type are available at the second point in time (step1906); selecting whether to display a particular item on an electronicvisual display in the retail store (step 1908), for example based on thedetermination of whether products of the particular product type areavailable at the first point in time and the determination of whetherproducts of the particular product type are available at the secondpoint in time; in response to a selection to display the particularitem, causing the electronic visual display to display the particularitem (step 1910); and in response to a selection not to display theparticular item, forgoing causing the electronic visual display todisplay the particular item (step 1912). In some examples, step 1902and/or step 1904 and/or step 1906 may be omitted from method 1900, thedetermination of whether products of the particular product type areavailable at the first point in time and/or the determination of whetherproducts of the particular product type are available at the secondpoint in time may be based on an analysis of data captured using aplurality of sensors positioned on the shelf and configured to bepositioned between the shelf and products positioned on the shelf (forexample as described in relation to FIG. 8A, 8B and 9), for example asdescribed herein.

FIG. 20 provides a flowchart of an exemplary method 2000 for customizedpresentation of items on electronic visual displays in retail stores,consistent with the present disclosure. In this example, method 2000 forcustomized presentation of items on electronic visual displays in retailstores may comprise: obtaining a plurality of images of products in aretail store captured using at least one image sensor (step 1902), theplurality of images may comprise at least a first image corresponding toa first point in time and a second image corresponding to a second pointin time, the first point in time is earlier than the second point intime; analyzing the first image to determine whether products of aparticular product type are available at the first point in time (step1904); analyzing the second image to determine whether products of theparticular product type are available at the second point in time (step1906); selecting at least one display parameter for a particular item(step 2008), for example based on the determination of whether productsof the particular product type are available at the first point in timeand the determination of whether products of the particular product typeare available at the second point in time; and using the selected atleast one display parameter to display the particular item on anelectronic visual display in the retail store (step 2010). In someexamples, step 1902 and/or step 1904 and/or step 1906 may be omittedfrom method 2000, the determination of whether products of theparticular product type are available at the first point in time and/orthe determination of whether products of the particular product type areavailable at the second point in time may be based on an analysis ofdata captured using a plurality of sensors positioned on the shelf andconfigured to be positioned between the shelf and products positioned onthe shelf (for example as described in relation to FIG. 8A, 8B and 9),for example as described herein.

FIG. 21 provides a flowchart of an exemplary method 2100 for selectingitems for presentation on electronic visual displays in retail stores,consistent with the present disclosure. In this example, method 2100 forselecting items for presentation on electronic visual displays in retailstores may comprise: obtaining an image of products in a retail storecaptured using at least one image sensor (step 2102); analyzing theimage to determine a condition of products of a particular product type(step 2104); selecting whether to display a particular item on anelectronic visual display in the retail store (step 2106), for examplebased on the determined condition of the products of the particularproduct type; in response to a selection to display the particular item,causing the electronic visual display to display the particular item(step 1910); and in response to a selection not to display theparticular item, forgoing causing the electronic visual display todisplay the particular item (step 1912).

FIG. 22 provides a flowchart of an exemplary method 2200 for customizedpresentation of items on electronic visual displays in retail stores,consistent with the present disclosure. In this example, method 2200 forcustomized presentation of items on electronic visual displays in retailstores may comprise: obtaining an image of products in a retail storecaptured using at least one image sensor (step 2102); analyzing theimage to determine a condition of products of a particular product type(step 2104); selecting at least one display parameter for a particularitem (step 2206), for example based on the determined condition of theproducts of the particular product type; and using the selected at leastone display parameter to display the particular item on an electronicvisual display in the retail store (step 2010).

Some non-limiting examples of the at least one display parameter (forexample, of method 2000, of method 2200, of step 2008, of step 2010, ofstep 2206, etc.) may include a display size for the particular item, amotion pattern for the particular item, a display position on theelectronic visual display for the particular item, a color scheme forthe particular item, a color scheme for a background of the particularitem, a brightness for the particular item, a contrast for theparticular item, a font for the particular item, a presentation time forthe particular item, and so forth.

Some non-limiting examples of the particular item (for example, ofmethod 1900, of method 2000, of method 2100, of method 2200, step 1908,step 1910, step 1912, step 2008, step 2010, step 2106, step 2206, etc.)may include an indication of the particular product type, a pricecorresponding to the particular product type, a name corresponding tothe particular product type, (such as a name of the particular producttype, a brand name corresponding to the particular product type, etc.),a promotion corresponding to the particular product type, a depiction ofat least part of a product of the particular product type, and so forth.In one example, the particular item of method 2100 and/or method 2200may include an indication of the condition of the products of theparticular product type, for example of the condition of the products ofthe particular product type determined by step 2104.

In some non-limiting examples, a particular product type may beconsidered available when products of the particular product type areavailable for sale in the retail store, when products of the particularproduct type are available for display in the retail store, whenproducts of the particular product type are present at selected locationwithin the retail store (for example, at a selected part of a shelf, ata selected shelf, at a selected part of a shelving unit, at a selectedshelving unit, at a select part of a display, at a selected display, ata selected part of a retail storage container, at a selected retailstorage container, etc.), and so forth.

In some examples, step 1902 may comprise obtaining a plurality of imagesof products in a retail store captured using at least one image sensor.The plurality of images obtained by step 1902 may comprise at least afirst image corresponding to a first point in time and a second imagecorresponding to a second point in time. The first point in time may beearlier than the second point in time. For example, at least part of theplurality of images may be read from memory (for example, from memory226 or from memory 1226), may be received from an external system (forexample, using network interface 206), may be captured using imagesensors (for example, using capturing device 125), and so forth.

In some examples, step 1904 may comprise analyzing the first imageobtained by step 1902 to determine whether products of a particularproduct type are available at the first point in time, and step 1906 maycomprise analyzing the second image obtained by step 1902 to determinewhether products of the particular product type are available at thesecond point in time. In some examples, the plurality of images obtainedby step 1902 may further comprise a preceding image corresponding to apreceding point in time, the preceding point in time may be earlier thanthe first point in time, and the preceding image may be analyzed todetermine whether products of the particular product type are availableat the preceding point in time. For example, a machine learning modelmay be trained using training examples to determine whether products ofa particular product type are available from an image, and the trainedmachine learning model may be used to analyze an image and determinewhether products of the particular product type are available at thepoint in time corresponding to the image. For example, step 1904 may usethe trained machine learning model to analyze the first image obtainedby step 1902 and to determine whether products of a particular producttype are available at the first point in time, step 1906 may use thetrained machine learning model to analyze the second image obtained bystep 1902 and to determine whether products of a particular product typeare available at the second point in time, and the trained machinelearning model may be used to analyze the preceding image and todetermine whether products of a particular product type are available atthe preceding point in time. An example of such training example mayinclude an image, together with a label indicating whether products of aselected product type are available. In another example, an artificialneural network (such as a deep neural network, a convolutional neuralnetwork, etc.) may be configured to determine whether products of aparticular product type are available from an image, and the artificialneural network may be used to analyze an image and determine whetherproducts of the particular product type are available at the point intime corresponding to the image.

In some examples, an electronic visual display (such as the electronicvisual display of method 1700, of method 1900, of method 2000, of method2100, of method 2200, of FIG. 16A-16F, etc.) may be connected to a shelfin the retail store. In one example, determining whether products of theparticular product type are available at the first point in time (forexample by step 1904) may include determining whether products of theparticular product type are available at the first point in time on theshelf, and/or determining whether products of the particular producttype are available at the second point in time (for example by step1906) may include determining whether products of the particular producttype are available at the second point in time on the shelf. In anotherexample, determining whether products of the particular product type areavailable at the first point in time (for example by step 1904) mayinclude determining whether products of the particular product type areavailable at the first point in time under the shelf, and/or determiningwhether products of the particular product type are available at thesecond point in time (for example by step 1906) may include determiningwhether products of the particular product type are available at thesecond point in time under the shelf

In some examples, an electronic visual display (such as the electronicvisual display of method 1700, of method 1800, of method 1900, of method2000, of method 2100, of method 2200, of FIG. 13A-13C, of FIG. 14A-14F,of FIG. 15A-15H, etc.) may be connected to a door of a retail storagecontainer in the retail store. In one example, determining whetherproducts of the particular product type are available at the first pointin time (for example by step 1904) may include determining whetherproducts of the particular product type are available at the first pointin time in the retail storage container, and/or determining whetherproducts of the particular product type are available at the secondpoint in time (for example by step 1906) may include determining whetherproducts of the particular product type are available at the secondpoint in time in the retail storage container.

Additionally or alternatively to step 1902, method 1900 and/or method2000 may comprise obtaining data captured at the first point in timeusing a plurality of sensors positioned on a shelf in the retail storeand configured to be positioned between the shelf and productspositioned on the shelf (for example as described in relation to FIG.8A, 8B and 9), and/or obtaining data captured at the second point intime using the plurality of sensors. Further, additionally oralternatively to step 1904, method 1900 and/or method 2000 may comprisebasing the determination of whether products of the particular producttype are available at the first point in time on an analysis of the datacaptured at the first point in time using the plurality of sensors.Further, additionally or alternatively to step 1906, method 1900 and/ormethod 2000 may comprise basing the determination of whether products ofthe particular product type are available at the second point in time onan analysis of the data captured at the second point in time using theplurality of sensors. Some non-limiting examples of such sensors mayinclude pressure sensors, touch sensors, light sensors, weight sensors,electrical impedance sensors, and so forth. For example, a machinelearning model may be trained using training examples to determinewhether products of the particular product type are available from datacaptured using the plurality of sensors, and the trained machinelearning model may be used to analyze the data captured at a particularpoint in time using the plurality of sensors to determine whetherproducts of the particular product type are available at the particularpoint in time. An example of such training example may include datacaptured using the plurality of sensors, together with a labelindicating whether products of the particular product type areavailable. In another example, an artificial neural network (such as adeep neural network, a convolutional neural network, etc.) may beconfigured to determine whether products of the particular product typeare available from data captured using the plurality of sensors, and theartificial neural network may be used to analyze the data captured at aparticular point in time using the plurality of sensors to determinewhether products of the particular product type are available at theparticular point in time. In one example, pressure data captured at thefirst point in time using a plurality of pressure sensors positioned ona shelf in the retail store and configured to be positioned between theshelf and products positioned on the shelf may be obtained, pressuredata captured at the second point in time using the plurality ofpressure sensors may be obtained, the determination of whether productsof the particular product type are available at the first point in timemay be based on an analysis of the pressure data captured at the firstpoint in time using the plurality of pressure sensors (for example asdescribed above), and the determination of whether products of theparticular product type are available at the second point in time may bebased on an analysis of the pressure data captured at the second pointin time using the plurality of pressure sensors (for example asdescribed above). In one example, touch data captured at the first pointin time using a plurality of touch sensors positioned on a shelf in theretail store and configured to be positioned between the shelf andproducts positioned on the shelf may be obtained, touch data captured atthe second point in time using the plurality of touch sensors may beobtained, the determination of whether products of the particularproduct type are available at the first point in time may be based on ananalysis of the touch data captured at the first point in time using theplurality of touch sensors (for example as described above), and thedetermination of whether products of the particular product type areavailable at the second point in time may be based on an analysis of thetouch data captured at the second point in time using the plurality oftouch sensors (for example as described above). In one example, lightdata captured at the first point in time using a plurality of lightsensors positioned on a shelf in the retail store and configured to bepositioned between the shelf and products positioned on the shelf may beobtained, light data captured at the second point in time using theplurality of light sensors may be obtained, the determination of whetherproducts of the particular product type are available at the first pointin time may be based on an analysis of the light data captured at thefirst point in time using the plurality of light sensors (for example asdescribed above), and the determination of whether products of theparticular product type are available at the second point in time may bebased on an analysis of the light data captured at the second point intime using the plurality of light sensors (for example as describedabove). In some examples, weight data captured at the first point intime using a weight sensor corresponding to at least part of a shelf inthe retail store may be obtained, weight data captured at the secondpoint in time using the weight sensor may be obtained, the determinationof whether products of the particular product type are available at thefirst point in time may be based on an analysis of the weight datacaptured at the first point in time using the weight sensor (for exampleas described above), and the determination of whether products of theparticular product type are available at the second point in time may bebased on an analysis of the weight data captured at the second point intime using the weight sensor (for example as described above). Forexample, the weight sensor may be configured to measure a weight of atleast one product placed on the shelf.

In some examples, step 1908 may comprise selecting whether to display aparticular item on an electronic visual display in the retail store, forexample based on the determination of whether products of the particularproduct type are available at the first point in time (for example ofstep 1904, based on the analysis of the data captured using theplurality of sensors positioned on a shelf in the retail store andconfigured to be positioned between the shelf and products positioned onthe shelf, etc.) and/or on the determination of whether products of theparticular product type are available at the second point in time (forexample of step 1906, based on the analysis of the data captured usingthe plurality of sensors positioned on a shelf in the retail store andconfigured to be positioned between the shelf and products positioned onthe shelf, and so forth). In one example, in response to a determinationthat products of the particular product type are missing at the firstpoint in time (for example by step 1904, based on the analysis of thedata captured using the plurality of sensors, etc.) and a determinationthat products of the particular product type are missing at the secondpoint in time (for example by step 1906, based on the analysis of thedata captured using the plurality of sensors, etc.), step 1908 mayselect not to display the particular item on the electronic visualdisplay in the retail store, and in response to at least one of adetermination that products of the particular product type are availableat the first point in time (for example by step 1904, based on theanalysis of the data captured using the plurality of sensors, etc.) anda determination that products of the particular product type areavailable at the second point in time (for example by step 1906, basedon the analysis of the data captured using the plurality of sensors,etc.), step 1908 may select to display the particular item on theelectronic visual display in the retail store, for example where theparticular item may include an indication of the particular producttype. In another example, in response to a determination that productsof the particular product type are missing at the first point in time(for example by step 1904, based on the analysis of the data capturedusing the plurality of sensors, etc.) and a determination that productsof the particular product type are missing at the second point in time(for example by step 1906, based on the analysis of the data capturedusing the plurality of sensors, etc.), step 1908 may select to displaythe particular item on the electronic visual display in the retailstore, and in response to at least one of a determination that productsof the particular product type are available at the first point in time(for example by step 1904, based on the analysis of the data capturedusing the plurality of sensors, etc.) and a determination that productsof the particular product type are available at the second point in time(for example by step 1906, based on the analysis of the data capturedusing the plurality of sensors, etc.), step 1908 may select not todisplay the particular item on the electronic visual display in theretail store, for example where the particular item may include anindication of a prolong shortage of the particular product type.

In some examples, the plurality of images obtained by step 1902 mayfurther comprise a preceding image corresponding to a preceding point intime, the preceding point in time may be earlier than the first point intime, and the preceding image may be analyzed to determine whetherproducts of the particular product type are available at the precedingpoint in time, for example as described above. Further, step 1908 mayfurther base the selection of whether to display the particular item onthe electronic visual display in the retail store on the determinationof whether products of particular product type are available at thepreceding point in time. In one example, in response to a determinationthat products of the particular product type are missing at thepreceding point in time, a determination that products of the particularproduct type are available at the first point in time (for example bystep 1904, based on the analysis of the data captured using theplurality of sensors, etc.) and a determination that products of theparticular product type are missing at the second point in time (forexample by step 1906, based on the analysis of the data captured usingthe plurality of sensors, etc.), step 1908 may select not to display theparticular item on the electronic visual display in the retail store,and in response to a determination that products of the particularproduct type are available at the preceding point in time, thedetermination that products of the particular product type are availableat the first point in time (for example by step 1904, based on theanalysis of the data captured using the plurality of sensors, etc.) andthe determination that products of the particular product type aremissing at the second point in time (for example by step 1906, based onthe analysis of the data captured using the plurality of sensors, etc.),step 1908 may select to display the particular item on the electronicvisual display in the retail store. In another example, in response to adetermination that products of the particular product type are missingat the preceding point in time, a determination that products of theparticular product type are missing at the first point in time (forexample by step 1904, based on the analysis of the data captured usingthe plurality of sensors, etc.) and a determination that products of theparticular product type are missing at the second point in time (forexample by step 1906, based on the analysis of the data captured usingthe plurality of sensors, etc.), step 1908 may select not to display theparticular item on the electronic visual display in the retail store,and in response to at least one of a determination that products of theparticular product type are available at the preceding point in time, adetermination that products of the particular product type are availableat the first point in time (for example by step 1904, based on theanalysis of the data captured using the plurality of sensors, etc.) andthe determination that products of the particular product type areavailable at the second point in time (for example by step 1906, basedon the analysis of the data captured using the plurality of sensors,etc.), step 1908 may select to display the particular item on theelectronic visual display in the retail store. In yet another example,in response to a determination that products of the particular producttype are missing at the preceding point in time, a determination thatproducts of the particular product type are missing at the first pointin time (for example by step 1904, based on the analysis of the datacaptured using the plurality of sensors, etc.) and a determination thatproducts of the particular product type are missing at the second pointin time (for example by step 1906, based on the analysis of the datacaptured using the plurality of sensors, etc.), step 1908 may select todisplay the particular item on the electronic visual display in theretail store, and in response to at least one of a determination thatproducts of the particular product type are available at the precedingpoint in time, a determination that products of the particular producttype are available at the first point in time (for example by step 1904,based on the analysis of the data captured using the plurality ofsensors, etc.) and the determination that products of the particularproduct type are available at the second point in time (for example bystep 1906, based on the analysis of the data captured using theplurality of sensors, etc.), step 1908 may select not to display theparticular item on the electronic visual display in the retail store,for example where the particular item may include an indication of aprolong shortage of the particular product type. In an additionalexample, in response to a determination that products of the particularproduct type are missing at the preceding point in time, a determinationthat products of the particular product type are available at the firstpoint in time (for example by step 1904, based on the analysis of thedata captured using the plurality of sensors, etc.) and a determinationthat products of the particular product type are missing at the secondpoint in time (for example by step 1906, based on the analysis of thedata captured using the plurality of sensors, etc.), step 1908 mayselect to display the particular item on the electronic visual displayin the retail store, and in response to at least one of a determinationthat products of the particular product type are available at thepreceding point in time and a determination that products of theparticular product type are available at the second point in time (forexample by step 1906, based on the analysis of the data captured usingthe plurality of sensors, etc.), step 1908 may select not to display theparticular item on the electronic visual display in the retail store,for example where the particular item may include an indication of arepeated shortage of the particular product type, or in another example,where in response to a determination that products of the particularproduct type are missing at the first point in time (for example by step1904, based on the analysis of the data captured using the plurality ofsensors, etc.), step 1908 may select not to display the particular itemon the electronic visual display in the retail store.

In some examples, step 1908 may further base the selection of whether todisplay the particular item on the electronic visual display in theretail store on an elapsed time between the first point in time and thesecond point in time. For example, in response to a first elapsed timebetween the first point in time and the second point in time, step 1908may select to display the particular item on the electronic visualdisplay in the retail store, and in response to a second elapsed timebetween the first point in time and the second point in time, step 1908may select not to display the particular item on the electronic visualdisplay in the retail store.

In some examples, step 1908 may further base the selection of whether todisplay the particular item on the electronic visual display in theretail store on an elapsed time since the second point in time. Forexample, in response to a first elapsed time since the second point intime, step 1908 may select to display the particular item on theelectronic visual display in the retail store, and in response to asecond elapsed time since the second point in time, step 1908 may selectnot to display the particular item on the electronic visual display inthe retail store.

In some examples, for example in response to a selection to display theparticular item by step 1908 and/or by step 2106, step 1910 may causethe electronic visual display to display the particular item, forexample as described above. In some examples, for example in response toa selection not to display the particular item by step 1908 and/or bystep 2106, step 1912 may forgo causing the electronic visual display todisplay the particular item.

In some examples, step 2008 may comprise selecting at least one displayparameter for a particular item, for example based on the determinationof whether products of the particular product type are available at thefirst point in time (for example of step 1904, based on the analysis ofthe data captured using the plurality of sensors positioned on a shelfin the retail store and configured to be positioned between the shelfand products positioned on the shelf, etc.) and/or the determination ofwhether products of the particular product type are available at thesecond point in time (for example of step 1906, based on the analysis ofthe data captured using the plurality of sensors positioned on a shelfin the retail store and configured to be positioned between the shelfand products positioned on the shelf, etc.) For example, in response toa first combination of the determination of whether products of theparticular product type are available at the first point in time and thedetermination of whether products of the particular product type areavailable at the second point in time, step 2008 may select a first atleast one display parameter for the particular item, and in response toa second combination of the determination of whether products of theparticular product type are available at the first point in time and thedetermination of whether products of the particular product type areavailable at the second point in time, step 2008 may select a second atleast one display parameter for the particular item, the at least onedisplay parameter may differ from the first at least one displayparameter.

In some examples, the plurality of images obtained by step 1902 mayfurther comprise a preceding image corresponding to a preceding point intime, the preceding point in time may be earlier than the first point intime, and the preceding image may be analyzed to determine whetherproducts of the particular product type are available at the precedingpoint in time, for example as described above. Further, step 2008 mayfurther base the selection of the at least one display parameter for theparticular item on the determination of whether products of theparticular product type are available at the preceding point in time.For example, in response to a first combination of the determination ofwhether products of the particular product type are available at thefirst point in time, the determination of whether products of theparticular product type are available at the second point in time andthe determination of whether products of the particular product type areavailable at the preceding point in time, step 2008 may select a firstat least one display parameter for the particular item, and in responseto a second combination of the determination of whether products of theparticular product type are available at the first point in time, thedetermination of whether products of the particular product type areavailable at the second point in time and the determination of whetherproducts of the particular product type are available at the precedingpoint in time, step 2008 may select a second at least one displayparameter for the particular item, the at least one display parametermay differ from the first at least one display parameter.

In some examples, step 2008 may further base the selection of the atleast one display parameter for the particular item on an elapsed timebetween the first point in time and the second point in time. Forexample, in response to a first elapsed time between the first point intime and the second point in time, step 2008 may select a first at leastone display parameter for the particular item, and in response to asecond elapsed time between the first point in time and the second pointin time, step 2008 may select a second at least one display parameterfor the particular item, the second at least one display parameter maydiffer from the first at least one display parameter.

In some examples, step 2008 may further base the selection of the atleast one display parameter for the particular item on an elapsed timesince the second point in time. For example, in response to a firstelapsed time since the second point in time, step 2008 may select afirst at least one display parameter for the particular item, and inresponse to a second elapsed time since the second point in time, step2008 may select a second at least one display parameter for theparticular item, the second at least one display parameter may differfrom the first at least one display parameter.

In some examples, step 2010 may comprise using the at least one displayparameter selected by step 2108 and/or by step 2206 to display theparticular item on an electronic visual display in the retail store.

In some examples, step 2102 may comprise obtaining an image of productsin a retail store captured using at least one image sensor. For example,the image of products in the retail store may be read from memory (forexample, from memory 226 or from memory 1226), may be received from anexternal system (for example, using network interface 206), may becaptured using image sensors (for example, using capturing device 125),and so forth.

In some examples, step 2104 may comprise analyzing the image obtained bystep 2102 to determine a condition of products of a particular producttype. In some examples, a preceding image of products in a retail storecaptured using the at least one image sensor at a preceding point intime before the capturing time of the image may be obtained, and thepreceding image to may be analyzed to determine a preceding condition ofthe products of the particular product type at the preceding point intime. For example, a machine learning model may be trained usingtraining examples to determine condition of products from images of theproducts, step 2104 may use the trained machine learning model toanalyze the image obtained by step 2102 to determine the condition ofproducts of the particular product type at the capturing time of theimage obtained by step 2102, and/or the trained machine learning modelmay be used to analyze the preceding image to determine the precedingcondition of the products of the particular product type at thepreceding point in time. An example of such training example may includean image of products, together with a label indicating the condition ofthe product. In another example, an artificial neural network (such as adeep neural network, a convolutional neural network, etc.) may beconfigured to determine condition of products from images of theproducts, step 2104 may use the artificial neural network to analyze theimage obtained by step 2102 to determine the condition of products ofthe particular product type at the capturing time of the image obtainedby step 2102, and/or the artificial neural network may be used toanalyze the preceding image to determine the preceding condition of theproducts of the particular product type at the preceding point in time.

In some examples, an electronic visual display (such as the electronicvisual display of method 1700, of method 1900, of method 2000, of method2100, of method 2200, of FIG. 16A-16F, etc.) may be connected to a shelfin the retail store. In one example, the condition of products of theparticular product type determined by step 2104 may include a conditionof products of the particular product type placed on the shelf. Inanother example, the condition of products of the particular producttype determined by step 2104 may include a condition of products of theparticular product type placed under the shelf

In some examples, an electronic visual display (such as the electronicvisual display of method 1700, of method 1800, of method 1900, of method2000, of method 2100, of method 2200, of FIG. 13A-13C, of FIG. 14A-14F,of FIG. 15A-15H, etc.) may be connected to a door of a retail storagecontainer in the retail store. In one example, the condition of productsof the particular product type determined by step 2104 may include acondition of products of the particular product type placed in theretail storage container.

Additionally or alternatively to step 2102, method 2100 and/or method2200 may comprise obtaining data captured using a plurality of sensorspositioned on a shelf in the retail store and configured to bepositioned between the shelf and products positioned on the shelf (forexample as described in relation to FIG. 8A, 8B and 9). Further,additionally or alternatively to step 2104, method 2100 and/or method2200 may comprise basing the determination of the condition of theproducts of the particular product type on an analysis of the datacaptured using the plurality of sensors. Some non-limiting examples ofsuch sensors may include pressure sensors, touch sensors, light sensors,weight sensors, electrical impedance sensors, and so forth. For example,a machine learning model may be trained using training examples todetermine a condition of the products of the particular product typefrom data captured using the plurality of sensors, and step 2104 may usethe trained machine learning model to analyze the data captured usingthe plurality of sensors to determine the condition of the products ofthe particular product type. An example of such training example mayinclude data captured using the plurality of sensors, together with alabel indicating the condition of the products of the particular producttype. In another example, an artificial neural network (such as a deepneural network, a convolutional neural network, etc.) may be configuredto determine a condition of the products of the particular product typefrom data captured using the plurality of sensors, and step 2104 may usethe artificial neural network to analyze the data captured using theplurality of sensors to determine the condition of the products of theparticular product type. In one example, electrical impedance datacaptured using a plurality of electrical impedance sensors positioned ona shelf in the retail store and configured to be positioned between theshelf and products positioned on the shelf may be obtained, and thedetermination of the condition of the products of the particular producttype may be based on an analysis of the electrical impedance datacaptured using the plurality of electrical impedance sensors (forexample as described above). In one example, light data captured using aplurality of light sensors positioned on a shelf in the retail store andconfigured to be positioned between the shelf and products positioned onthe shelf may be obtained, and the determination of the condition of theproducts of the particular product type may be based on an analysis ofthe light data captured using the plurality of light sensors (forexample as described above).

In some examples, step 2106 may comprise selecting whether to display aparticular item on an electronic visual display in the retail store, forexample based on the condition of the products of the particular producttype determined by step 2104, based on the condition of the products ofthe particular product type determined based on the analysis of the datacaptured using the plurality of sensors positioned on a shelf in theretail store and configured to be positioned between the shelf andproducts positioned on the shelf, and so forth. For example, in responseto a first determined condition of the products of the particularproduct type, step 2106 may select to display the particular item on theelectronic visual display in the retail store, and in response to asecond determined condition of the products of the particular producttype, step 2106 may select not to display the particular item on theelectronic visual display in the retail store. In another example, inresponse to a first determined condition of the products of theparticular product type, step 2106 may select to display the particularitem on the electronic visual display in the retail store, and inresponse to a second determined condition of the products of theparticular product type, step 2106 may select to display an alternativeitem on the electronic visual display in the retail store. In yetanother example, in response to a determination that the condition ofthe products of the particular product type is a good condition, step2106 may select to display the particular item on the electronic visualdisplay in the retail store, and in response to a determination that thecondition of the products of the particular product type is a badcondition, step 2106 may select not to display the particular item onthe electronic visual display in the retail store, for example where theparticular item may include an indication of the particular producttype. In an additional example, in response to a determination that thecondition of the products of the particular product type is a badcondition, step 2106 may select to display the particular item on theelectronic visual display in the retail store, and in response to adetermination that the condition of the products of the particularproduct type is a good condition, step 2106 may select not to displaythe particular item on the electronic visual display in the retailstore, for example where the particular item may include a promotioncorresponding to the particular product type. In some examples, inresponse to a determination that the condition of the products of theparticular product type is a condition that requires maintenance, step2106 may select to display the particular item on the electronic visualdisplay in the retail store, and in response to a determination that thecondition of the products of the particular product type is a conditionthat do not require maintenance, step 2106 may select not to display theparticular item on the electronic visual display in the retail store,for example where the particular item may include an indication of therequired maintenance, may include an indication of the condition, mayinclude an indication to a store associate, and so forth.

In some examples, step 2106 may further base the selection of whether todisplay the particular item on the electronic visual display in theretail store on an elapsed time since the capturing of the imageobtained by step 2102. For example, in response to a first elapsed timesince the capturing of the image obtained by step 2102, step 2106 mayselect to display the particular item on the electronic visual displayin the retail store, and in response to a second elapsed time since thecapturing of the image obtained by step 2102, step 2106 may select notto display the particular item on the electronic visual display in theretail store.

In some examples, a preceding image of products in a retail storecaptured using the at least one image sensor at a preceding point intime before the capturing time of the image may be obtained, and thepreceding image to may be analyzed to determine a preceding condition ofthe products of the particular product type at the preceding point intime. Further, step 2106 may further base the selection of whether todisplay the particular item on the electronic visual display in theretail store on the determined preceding condition of the products ofthe particular product type at the preceding point in time. For example,in response to a first determined preceding condition, step 2106 mayselect to display the particular item on the electronic visual displayin the retail store, and in response to a second determined precedingcondition, step 2106 may select not to display the particular item onthe electronic visual display in the retail store. In some examples, thedetermined preceding condition may be compared with the determinedcondition example, and step 2106 may base the selection of whether todisplay the particular item on the electronic visual display in theretail store on a result of the comparison. For example, in response toa first result of the comparison, step 2106 may select to display theparticular item on the electronic visual display in the retail store,and in response to a second result of the comparison, step 2106 mayselect not to display the particular item on the electronic visualdisplay in the retail store. In some examples, the determined precedingcondition and the determined condition may be used to predict a futurecondition of products of the particular product type at a later point intime after the capturing time of the image (for example, using anextrapolation algorithm), and step 2106 may base the selection ofwhether to display the particular item on the electronic visual displayin the retail store on the predicted future condition. For example, inresponse to a first predicted future condition, step 2106 may select todisplay the particular item on the electronic visual display in theretail store, and in response to a second predicted future condition,step 2106 may select not to display the particular item on theelectronic visual display in the retail store.

In some examples, the image obtained by step 2102 may be analyzed (forexample in a similar manner as described above with respect to step2104) to determine a condition of the products of a second product type(the second product type may differ from the particular product type),and step 2106 may further base the selection of whether to display theparticular item on the electronic visual display in the retail store onthe determined condition of the products of the second product type. Forexample, the determined condition of the products of the particularproduct type may be compared with the determined condition of theproducts of the second product type, in response to a first result ofthe comparison, step 2106 may select to display the particular item onthe electronic visual display in the retail store, and in response to asecond result of the comparison, step 2106 may select not to display theparticular item on the electronic visual display in the retail store.

In some examples, the selection of whether to display the particularitem on the electronic visual display in the retail store by step 1908and/or by step 2106 may be further based on information related to aperson in a vicinity of the electronic visual display. For example, inresponse to a first information related to the person in the vicinity ofthe electronic visual display, step 1908 and/or step 2106 may select todisplay the particular item on the electronic visual display in theretail store, and in response to a second information related to theperson in the vicinity of the electronic visual display, step 1908and/or step 2106 may select not to display the particular item on theelectronic visual display in the retail store. In some examples, theselection of whether to display the particular item on the electronicvisual display in the retail store by step 1908 and/or by step 2106 maybe further based on an analysis of an image of a person in a vicinity ofthe electronic visual display. For example, the image may be analyzed todetermine information related to the person (such as an identity of theperson, an indication of a gender of the person, an indication of an ageof a person, an indication of a social economic group of the person, aheight of the person, an indication of a weight of the person, etc.),and the selection of whether to display the particular item on theelectronic visual display in the retail store by step 1908 and/or bystep 2106 may be further based on the determined information related tothe person. In another example, the selection of whether to display theparticular item on the electronic visual display in the retail store bystep 1908 and/or by step 2106 may be further based on an identity of aperson in a vicinity of the electronic visual display. In yet anotherexample, the selection of whether to display the particular item on theelectronic visual display in the retail store by step 1908 and/or bystep 2106 may be further based on at least one of an indication of agender of the person, an indication of an age of the person, and anindication of a social economic group of the person. In an additionalexample, the selection of whether to display the particular item on theelectronic visual display in the retail store by step 1908 and/or bystep 2106 may be further based on at least one of an indication of aheight of the person and an indication of a weight of the person.

In some examples, the selection of whether to display the particularitem on the electronic visual display in the retail store by step 1908and/or by step 2106 may be further based on a current time of day and/oron opening hours of the retail store. For example, in response to afirst time of day, step 1908 and/or step 2106 may select to display theparticular item on the electronic visual display in the retail store,and in response to a second time of day, step 1908 and/or step 2106 mayselect not to display the particular item on the electronic visualdisplay in the retail store. In another example, the current time of daymay be compared with opening hours of the retail store in response to afirst result of the comparison, step 1908 and/or step 2106 may select todisplay the particular item on the electronic visual display in theretail store, and in response to a second result of the comparison, step1908 and/or step 2106 may select not to display the particular item onthe electronic visual display in the retail store.

In some examples, step 2206 may comprise selecting at least one displayparameter for a particular item, for example based on the condition ofthe products of the particular product type determined by step 2104,based on the condition of the products of the particular product typedetermined based on the analysis of the data captured using theplurality of sensors positioned on a shelf in the retail store andconfigured to be positioned between the shelf and products positioned onthe shelf, and so forth. For example, in response to a first determinedcondition of the products of the particular product type, step 2206 mayselect a first at least one display parameter for the particular item,and in response to a second determined condition of the products of theparticular product type, step 2206 may select a second at least onedisplay parameter for the particular item, the second at least onedisplay parameter may differ from the first at least one displayparameter. In another example, in response to a determination that thecondition of the products of the particular product type is a goodcondition, step 2206 may select a first at least one display parameterfor the particular item, and in response to a determination that thecondition of the products of the particular product type is a badcondition, step 2206 may select a second at least one display parameterfor the particular item, the second at least one display parameter maydiffer from the first at least one display parameter. In yet anotherexample, in response to a determination that the condition of theproducts of the particular product type is a condition that requiresmaintenance, step 2206 may select a first at least one display parameterfor the particular item, and in response to a determination that thecondition of the products of the particular product type is a conditionthat do not require maintenance, step 2206 may select a second at leastone display parameter for the particular item, the second at least onedisplay parameter may differ from the first at least one displayparameter.

In some examples, the determined condition of the products of theparticular product type may be a condition that requires maintenance,and the image obtained by step 2102 may be analyzed to determine anindicator of urgency of the required maintenance. For example, a machinelearning model may be trained using training examples to determineurgency of required maintenance from images, and the trained machinelearning model may be used to analyze the image obtained by step 2102and determine the indicator of urgency of the required maintenance. Anexample of such training example may include an image of a conditionrequiring maintenance activity, together with a label indicating therequired maintenance. In another example, an artificial neural network(such as a deep neural network, a convolutional neural network, etc.)may be configured to determine urgency of required maintenance fromimages, and the artificial neural network may be used to analyze theimage obtained by step 2102 and determine the indicator of urgency ofthe required maintenance. In yet another example, the indicator ofurgency of the required maintenance may be determined based on thedetermined condition of the products of the particular product type, forexample using a lookup table or a function that takes as input thedetermined condition of the products of the particular product type andreturns a corresponding indication of urgency. Further, in someexamples, step 2206 may further base the selection of the at least onedisplay parameter for the particular item on the determined indicator ofthe urgency of the required maintenance, for example where theparticular item may include an indication of the required maintenance,may include an indication of the condition, may include an indication toa store associate, and so forth. For example, in response to a firstdetermined indicator of the urgency, step 2206 may select a first atleast one display parameter for the particular item, and in response toa second determined indicator of the urgency, step 2206 may select asecond at least one display parameter for the particular item, thesecond at least one display parameter may differ from the first at leastone display parameter.

In some examples, step 2206 may further base the selection of the atleast one display parameter for the particular item on an elapsed timesince the capturing of the image obtained by step 2102. For example, inresponse to a first elapsed time since the capturing of the imageobtained by step 2102, step 2206 may select a first at least one displayparameter for the particular item, and in response to a second elapsedtime since the capturing of the image obtained by step 2102, step 2206may select a second at least one display parameter for the particularitem, the second at least one display parameter may differ from thefirst at least one display parameter.

In some examples, a preceding image of products in a retail storecaptured using the at least one image sensor at a preceding point intime before the capturing time of the image may be obtained, and thepreceding image to may be analyzed to determine a preceding condition ofthe products of the particular product type at the preceding point intime. Further, step 2206 may further base the selection of the at leastone display parameter for the particular item on the determinedpreceding condition. For example, in response to a first determinedpreceding condition, step 2206 may select a first at least one displayparameter for the particular item, and in response to a seconddetermined preceding condition, step 2206 may select a second at leastone display parameter for the particular item, the second at least onedisplay parameter may differ from the first at least one displayparameter. In some examples, the determined preceding condition may becompared with the determined condition example, and step 2206 may basethe selection of the at least one display parameter for the particularitem on a result of the comparison. For example, in response to a firstresult of the comparison, step 2206 may select a first at least onedisplay parameter for the particular item, and in response to a secondresult of the comparison, step 2206 may select a second at least onedisplay parameter for the particular item, the second at least onedisplay parameter may differ from the first at least one displayparameter. In some examples, the determined preceding condition and thedetermined condition may be used to predict a future condition ofproducts of the particular product type at a later point in time afterthe capturing time of the image (for example, using an extrapolationalgorithm), and step 2206 may base the selection of the at least onedisplay parameter for the particular item on the electronic visualdisplay in the retail store on the predicted future condition. Forexample, in response to a first predicted future condition, step 2206may select a first at least one display parameter for the particularitem, and in response to a second predicted future condition, step 2206may select a second at least one display parameter for the particularitem, the second at least one display parameter may differ from thefirst at least one display parameter.

In some examples, the image obtained by step 2102 may be analyzed (forexample in a similar manner as described above with respect to step2104) to determine a condition of the products of a second product type(the second product type may differ from the particular product type),and step 2206 may further base the selection of the at least one displayparameter for the particular item on the determined condition of theproducts of the second product type. For example, the determinedcondition of the products of the particular product type may be comparedwith the determined condition of the products of the second producttype, in response to a first result of the comparison, step 2206 mayselect a first at least one display parameter for the particular item,and in response to a second result of the comparison, step 2206 mayselect a second at least one display parameter for the particular item,the second at least one display parameter may differ from the first atleast one display parameter.

In some examples, the selection of the at least one display parameterfor the particular item by step 2008 and/or by step 2206 may be furtherbased on information related to a person in a vicinity of the electronicvisual display. For example, in response to a first information relatedto the person in the vicinity of the electronic visual display, step2008 and/or by step 2206 may select a first at least one displayparameter for the particular item, and in response to a secondinformation related to the person in the vicinity of the electronicvisual display, step 2008 and/or by step 2206 may select a second atleast one display parameter for the particular item, the second at leastone display parameter for the particular item may differ from the firstat least one display parameter for the particular item. In someexamples, the selection of the at least one display parameter for theparticular item by step 2008 and/or by step 2206 may be further based onan analysis of an image of a person in a vicinity of the electronicvisual display, the image may be captured from an environment of theelectronic visual display using an image sensor. For example, the imagemay be analyzed to determine information related to the person (such asan identity of the person, an indication of a gender of the person, anindication of an age of a person, an indication of a social economicgroup of the person, a height of the person, an indication of a weightof the person, etc.), and the selection of the at least one displayparameter for the particular item by step 2008 and/or by step 2206 maybe further based on the determined information related to the person. Inanother example, the selection of the at least one display parameter forthe particular item by step 2008 and/or by step 2206 may be furtherbased on an identity of a person in a vicinity of the electronic visualdisplay. In yet another example, the selection of the at least onedisplay parameter for the particular item by step 2008 and/or by step2206 may be further based on at least one of an indication of a genderof the person, an indication of an age of the person, and an indicationof a social economic group of the person. In an additional example, theselection of the at least one display parameter for the particular itemby step 2008 and/or by step 2206 may be further based on at least one ofan indication of a height of the person and an indication of a weight ofthe person.

In some examples, the selection of the at least one display parameterfor the particular item by step 2008 and/or by step 2206 may be furtherbased on a current time of day and/or on opening hours of the retailstore. For example, in response to a first time of day, step 2008 and/orby step 2206 may select a first at least one display parameter for theparticular item, and in response to a second time of day, step 2008and/or by step 2206 may select a second at least one display parameterfor the particular item, the second at least one display parameter forthe particular item may differ from the first at least one displayparameter for the particular item. In another example, the current timeof day may be compared with opening hours of the retail store inresponse to a first result of the comparison, step 2008 and/or by step2206 may select a first at least one display parameter for theparticular item, and in response to a second result of the comparison,step 2008 and/or by step 2206 may select a second at least one displayparameter for the particular item, the second at least one displayparameter for the particular item may differ from the first at least onedisplay parameter for the particular item.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, 4K Ultra HD Blu-ray,or other optical drive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

1-106. (canceled)
 107. A non-transitory computer-readable mediumincluding instructions that when executed by a processor cause theprocessor to perform a method for customized presentation of items onelectronic visual displays in retail stores, the method comprising:obtaining an image of products in a retail store captured using at leastone image sensor; analyzing the image to determine a condition ofproducts of a particular product type; based on the determined conditionof the products of the particular product type, selecting at least onedisplay parameter for a particular item; and using the selected at leastone display parameter to display the particular item on an electronicvisual display in the retail store.
 108. The non-transitorycomputer-readable medium of claim 107, wherein the at least one displayparameter includes a display size for the particular item.
 109. Thenon-transitory computer-readable medium of claim 107, wherein the atleast one display parameter includes a motion pattern for the particularitem.
 110. The non-transitory computer-readable medium of claim 107,wherein the at least one display parameter includes a display positionon the electronic visual display for the particular item.
 111. Thenon-transitory computer-readable medium of claim 107, wherein the atleast one display parameter includes a color scheme for the particularitem.
 112. The non-transitory computer-readable medium of claim 107,wherein the selection of the at least one display parameter for theparticular item is further based on an elapsed time since the capturingof the image.
 113. The non-transitory computer-readable medium of claim107, wherein the selection of the at least one display parameter for theparticular item is further based on a time of day.
 114. Thenon-transitory computer-readable medium of claim 107, wherein theselection of the at least one display parameter for the particular itemis further based on information related to a person in a vicinity of theelectronic visual display.
 115. The non-transitory computer-readablemedium of claim 107, wherein the method further comprises: obtaining apreceding image of products in a retail store captured using the atleast one image sensor at a preceding point in time before the capturingtime of the image; analyzing the preceding image to determine apreceding condition of the products of the particular product type atthe preceding point in time; and further basing the selection of the atleast one display parameter for the particular item on the determinedpreceding condition.
 116. The non-transitory computer-readable medium ofclaim 115, wherein the method further comprises: comparing thedetermined preceding condition with the determined condition; and basingthe selection of the at least one display parameter for the particularitem on a result of the comparison.
 117. The non-transitorycomputer-readable medium of claim 115, wherein the method furthercomprises: using the determined preceding condition and the determinedcondition to predict a future condition of products of the particularproduct type at a later point in time after the capturing time of theimage; and basing the selection of the at least one display parameterfor the particular item on the predicted future condition.
 118. Thenon-transitory computer-readable medium of claim 107, wherein theelectronic visual display is connected to a shelf in the retail store.119. The non-transitory computer-readable medium of claim 107, whereinthe electronic visual display is connected to a door of a retail storagecontainer in the retail store.
 120. The non-transitory computer-readablemedium of claim 107, wherein the electronic visual display is part of apersonal device of a store associate.
 121. The non-transitorycomputer-readable medium of claim 107, wherein the electronic visualdisplay is part of a personal device of a customer.
 122. Thenon-transitory computer-readable medium of claim 107, wherein the methodfurther comprises: obtaining data captured using a plurality of sensorspositioned on a shelf in the retail store and configured to bepositioned between the shelf and products positioned on the shelf; andbasing the determination of the condition of the products of theparticular product type on an analysis of the data captured using theplurality of sensors.
 123. The non-transitory computer-readable mediumof claim 107, wherein the determined condition of the products of theparticular product type is a condition that requires maintenance, andthe method further comprises: analyzing the image to determine anindicator of urgency of the required maintenance; and basing theselection of the at least one display parameter for the particular itemon the determined indicator of urgency.
 124. The non-transitorycomputer-readable medium of claim 107, wherein the method furthercomprises: analyzing the image to determine a condition of the productsof a second product type, the second product type differs from theparticular product type; and further basing the selection of the atleast one display parameter for the particular item on the determinedcondition of the products of the second product type.
 125. A method forcustomized presentation of items on electronic visual displays in retailstores, the method comprising: obtaining an image of products in aretail store captured using at least one image sensor; analyzing theimage to determine a condition of products of a particular product type;based on the determined condition of the products of the particularproduct type, selecting at least one display parameter for a particularitem; and using the selected at least one display parameter to displaythe particular item on an electronic visual display in the retail store.126. A system for customized presentation of items on electronic visualdisplays in retail stores, the system comprising: at least one processorconfigured to: obtain an image of products in a retail store capturedusing at least one image sensor; analyze the image to determine acondition of products of a particular product type; based on thedetermined condition of the products of the particular product type,select at least one display parameter for a particular item; and use theselected at least one display parameter to display the particular itemon an electronic visual display in the retail store.